Class-Distribution-Aware Pseudo-Labeling for Semi-Supervised Multi-Label Learning
Authors: Ming-Kun Xie
Jia-Hao Xiao
Hao-Zhe Liu
Gang Niu
Masashi Sugiyama
Sheng-Jun Huang
Venue: NeurIPS 2023 poster (2023)
Keywords: Semi-supervised multi-label learning
pseudo labeling.

Abstract: Pseudo-labeling has emerged as a popular and effective approach for utilizing unlabeled data. However, in the context of semi-supervised multi-label learning (SSMLL), conventional pseudo-labeling methods encounter difficulties when dealing with instances associated with multiple labels and an unknown label count. These limitations often result in the introduction of false positive labels or the neglect of true positive ones. To overcome these challenges, this paper proposes a novel solution called Class-Aware Pseudo-Labeling (CAP) that performs pseudo-labeling in a class-aware manner. The proposed approach introduces a regularized learning framework incorporating class-aware thresholds, which effectively control the assignment of positive and negative pseudo-labels for each class. Notably, even with a small proportion of labeled examples, our observations demonstrate that the estimated class distribution serves as a reliable approximation. Motivated by this finding, we develop a class-distribution-aware thresholding strategy to ensure the alignment of pseudo-label distribution with the true distribution. The correctness of the estimated class distribution is theoretically verified, and a generalization error bound is provided for our proposed method. Extensive experiments on multiple benchmark datasets confirm the efficacy of CAP in addressing the challenges of SSMLL problems.

Class-Distribution-AAarr Pseudo-Labbin for Semi-Supervised lulli Learning
Ming-Kun Xiel. Jia-Hao Xiao 1.2, Hao-Zhe Liu Gang Niu Masashi Sugiyama
Sheng-Jun Huang
Nanjing University of Aeronautics and Astronautics
MIIT Key Laboratory Pattern Analysis and Machine Intelligence, Nanjing, China
'RIKEN Center for Advanced Intelligence Project "The University of Tokyo, Tokyo, Japan
{mkxie, jiahaoxiao, haozheliu, juangsj/enuas edu. cn gang. miu. falummm@mmail. com com sugi@k. u-tokyo.ace ac.
Abstract
Pseudo-labeling has emerged popular and effective approach for utilizing unlabeled data. However, the context semi-supervised multi-label learning (SSMLL), conventional pseudo-labeling methods encounter difficulties when deal- ing with instances associated with multiple labels and an unknown label count. These limitations often result in the introduction of false positive labels the neglect true positive ones. overcome these challenges, this paper proposes a novel solution called Class-Award Pseudo-Labeling (CAP) that performs pseudo- labeling in a aware manner. The proposed approach introduces a regularized learning framework incorporating class-aware thresholds, which effectively control the assignment of positive and negative pseudo-labels for each class. Notably, even with small proportion of labeled examples, observations demonstrate that the estimated class distribution serves as a reliable approximation. Moti- vated by this finding, we develop class-distribution-aaaa thresholding strategy to ensure the alignment of pseudo-label distribution with the true distribution. The correctness of the estimated class distribution is theoretically verified, and a generalization error bound is provided for our proposed method. Extensive experiments on multiple benchmark datasets confirm the efficacy of CAP in ad- dressing the challenges of SSMLL problems. The implementation is available at com www/milkkee Settxii//SMMLLCAA
Introduction
In single-label supervised learning, each instance is assumed to be associated with only class label, while many realistic scenarios may be multi-labeled, where each instance consists of multiple semantics. For example, image of a nature landscape often contains the objects of sky, cloud, and mountain. Multi-label learning (MLL) a practical and effective paradigm for handling examples with multiple labels. It trains classifier that can predict all the relevant labels for unseen instances based on the given training examples. A large number of recent works have witnessed the great progress that MIL has made in many practical applications (27). e.g., image annotation (1]. protein
subcellular localization (28), and visual attribute recognition 32
Thanks to its powerful capacity, the deep neural network (DNN) has become prevalent learning model for handling MLL examples (34). Unfortunately. it requires a large number of precisely labeled
"Correspondence to: Sheng-Jur Jun Huang (huangsj@ maa.jj/uuaaaauuunn.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

examples to achieve favorable performance. This leads to a high cost of manual annotation, especially when the dataset is large and the labeling task must be carried out by an expert. Given that it is hard to train an effective DNN based on small subset of training examples, it is rather important to exploit the information from unlabeled instances. The problem has been formalized as a learning framework called remi-supervised multi-label learning (SSMLL), which aims to train a classifier based on a small set of labeled MLL examples and a large set of unlabeled ones.
Compared to semi-supervised learning (SSL) that has made great progress received relatively less attention in the context of deep learning. Generally, there are still three main challenges towards the development of SSMLL. Firstly, since each instance is associated with multiple labels and the number is unknown, the commonly used pseudo-labeling strategy that selects the most probable label or the top-k probable labels cannot be applied to the SSMLL problems. It would face the dilemma of either introducing false positive labels or neglecting true positive ones. Secondly, due to the intrinsic class-imbalance property of MLL data, it is hard to achieve favorable performance by using fixed threshold for each instance. Thirdly, recent studies have mainly focused on multi-label learning with missing labels (MLML) [1] 2 scenarios, where each training instance is assumed to be assigned with a subset of true labels. Unfortunately, these methods often fail to achieve favorable performance, or cannot even be applied to the SSMLL scenarios, since most of them were designed under the assumption of MLML.
SSMLL has 38
To solve these challenges, in this paper, we propose a novel Class- Pseudo-labeling (CAP) method for handling the SSMLL problems. Unlike the existing methods, we perform pseudo-labeling in class-aware manner to avoid estimating the number of true labels for each instance, which can be very hard in practice. Specifically, a regularized learning framework is proposed to determine the numbers of positive and negative pseudo-labels for each class based on the class-aware thresholds. Given that the true class distribution is unknown, we alternatively determine the thresholds based on the estimated class distribution of labeled data, which can be tight approximation according to our observation. Our theoretical results show the correctness of estimated class distribution and provide a generalization error bound for CAP. Extensive experimental results on multiple benchmark datasets with a variety of comparing methods validate that the proposed method can achieve state-of- the-art art performance.
2 Related Work
Thanks to the powerful learning capacity of DNNs, MLL has made great advances in the context of deep learning. Some methods designed architectures [6] or training strategies [22 49 to exploit the label correlations. Some other methods designed sophisticated loss functions to improve the performance of MIL 34. The last group of methods designed specific architectures to capture the objects related to semantic labels. Global-average- peoooiggpoooling (GAP) based models and attention- based models 22 26 are two groups of representative methods.
There are relatively few works that study how to improve the performance of deep models in SSMLL scenarios. Instead of end-to-end training, the only deep SSMLL method 43) performed the two- stage training, which first used a DNN to extract features, and then used a linear model to perform classification. (37] proposed deep sequential generative model to handle the noisy labels collected by crowdsoureing and unlabeled data simultaneously. 20 focused on the transductive and non non-deep scenario, and thus cannot be applied to our setting. The method proposed by 33 utilized the graph neural network (GNN) to deal with SSMLL data with graph structures. In contrast, there are many works that trained linear models to solve the SSMLL problems [5] 15 42
52
50
40). M3DN was
proposed to deal with SSMLL data in multi-modal multi-instance learning scenarios by adopting optimal transport technique [48)
Pseudo-labeling has become a popular method in semi-supervised learning (SSL). The idea was firstly applied to semi-supervised training of deep neural networks 233. Subsequently, a great deal of works have been devoted to improving the quality of pseudo-labels either by adopting consistency regularization 4 [38, or by using distribution alignment 30 33 The contrastive learning technique
has been applied to improve the performance of SSL 24 To improve the reliability of pseudo-
labels, an uncertainty-awal pseudo-labeling method proposed in
[35 selected reliable pseudo-labels
based on the prediction uncertainty. Recent studies have also paid attention to dealing with the
problem of pseudo-labeling in SSL scenarios [18 45 [4] Unlike FixMatch that

---

selects unlabeled examples with fixed threshold, Dash 47 selected unlabeled examples with dynamic threshold, with the goal of achieving better pseudo-labeling performance. FlexMatch [5T] was proposed to select unlabeled examples for every class according to the current learning status of the model. Several works have been explored the idea of selecting different thresholds for different
classes to improve the performance of SLL [13 14 44 However, these methods are designed for the multi-class single-label scenario, and cannot be directly applied to the multi-label scenario.
In order to reduce the annotation cost, a cost-effective strategy is to assign a subset of true labels to each instance. For example, [[]] designed partial binary cross entropy (BCE) loss that re-weights the losses of known labels. As an extreme case of MLML, single positive multi-label learning (SPML) [8] 41] assumes that only one of multiple true labels can be observed during the training stage. The pioneering work [8] trains DNNs by simply treating unobserved labels as negative ones and utilizes the regularization to alleviate the harmfulness of false negative labels.
[53 propose
asymmetric pseudo labeling technique to recover true labels.
3 The Method
In the SSMLL problem, let æ E X' be a feature vector and y E y be its corresponding label vector, where X' = Hd is the feature space and D {0, 1 is the label space with g possible class labels. Here, y/- = indicates the k-th label is relevant to the instance, while yk = 0, otherwise. Suppose that we are given labeled dataset with n training examples D; {( y;)}? and an unlabeled dataset with m training instances D. = {x, 1/m Our goal is to train a DNN f(x: 8) based on the labeled dataset D; and unlabeled dataset D.., where A is the parameter of the network. For notational simplicity, we omit the notation 0 and let f(x) be the predicted probability distribution over classes
and (xr) be the predicted probability of the k-th class for input a:
Typical multi-label learning methods usually train a DNN with the commonly used binary cross entropy (BCE) loss, which decomposes the original task into multiple binary classification problems. Unfortunately, BCE loss often suffers from positive-negatio
imbalance issue. To mitigate this problem, we adopt the asymmetric loss (ASL) 34 which is a variant of focal loss with different focusing parameters for positive and negative instances. In our experiment, we found it works better than BCE loss. Formally, given the predicted probabilities f(x on instance ε, the ASL loss is defined as
(kll(f(())) + (1 yk 3)) (z) ((f(z),)==
(I)
Here, (f()) -11 fa) log) and (o(f))) = ( log(1 (1 - fk) represent the losses calculated on positive and negative labels, where A and Ao are positive and negative focusing parameters.
3.1 Instance-Aware Pseudo-Labeling
The loss function may not be the best choice solve the SSMLL problem, since besides the labeled training examples, there still exist a large number of unlabeled training examples. To exploit the information of unlabeled data, inspired by recent SSL works 38) an intuitive strategy is assigning the unlabeled instances with pseudo-labels based on the model outputs. Formally, we define the unlabeled loss C. as
(.(f)),,) D k=1 (cll(f + (1- (c)
where 9 10...... 9 represents the pscudo-label vector for instance
In the above formulation, the most significant element is how to obtain the pseudo-labels ý that significantly affects the final performance of SSMLL. Most of existing pseudo-labeling methods are performed in an instance-aware manner by assigning pseudo-label labels to each unlabeled instance based on its probability distribution. Below, we briefly review three instance-aware pseudo-labeling strategies that can be applied to the SSMLL problems. The most commonly used strategy adopted by the SSL method called FixMatch (8 is to select the most probable label as the ground-truth one:
{
1 if l = arg max fc(x),
0x
cEle]
0 otherwise.
3

---

One advantage of the strategy is that it is likely to safely identify a true label for each unlabeled training instance. Unfortunately, it is obvious that the strategy would neglect multiple true labels. Generally, it transforms the unlabeled dataset into another learning scenario called single positíve multi-label learning (SPML) [ where only one of multiple positive labels is available for each instance. A straightforward strategy is to simply treat unobserved labels as negative ones. Although this strategy enables us to train a classifier based on SPML data, it would introduce a large number of false negative labels, leading to unfavorable performance.
The second choice is an improved version of the above strategy, which selects the top probable labels as the true ones:
1 if fx(e)) 2
Úk 0 otherwise,
where 7' is the I-th predicted probability in a descending order. The strategy conducts competition among labels, and selects top winners. The optimal solution is to set as the true number of positive labels for each unlabeled instance. Unfortunately, since the number is unknown in practice, as a compromise, we set as the average number of positive labels per instance. Given that the true number does not always equal to the average number, it would be caught in a dilemma of either introducing false positive labels or neglecting true positive ones.
The last choice is to adopt instance-aware threshold Ty that separates positive and negative labels for each unlabeled instance.
Øرز {1 if fx(z))
otherwise.
Compared to the above methods, this strategy achieves strong flexibility that allows it to assign different numbers of positive labels to different instances. A potential limitation is that hard to find the optimal thresholds for different instances. In practice, feasible solution is adopt global threshold, that is Vj E [m), T = т. Obviously, it is impossible to adopt global threshold that is optimal for all instances, especially considering the class-imbalance property of MLL data. In general, a large threshold often leads to a small recall score of tail classes, which indicates that less positive labels would be identified. While small threshold often results in small precision score of head classes, which indicates besides positive labels, a great deal of negative ones would be treated as positive ones. The dilemma prevents the model from obtaining favorable performance.
3.2 Class-Aware Pseudo-Labeling
As discussed above, in many real-world scenarios, it is really difficult to acquire the true number of positive labels for each instance. This leads the instance-aware pseudo-labeling methods to be caught in the dilemma of either mislabeling false positive labels or neglecting true positive labels, resulting noticeable decrease of the model performance.
To solve this issue, we propose a regularized learning framework to assign pseudo-labels in a class-aware manner. Formally, reformulate the optimization problem of SSMLL
min 8.0
Vinl (fa (1)
9
Ú5k E (Jk(m;)) + (ox
akūs + 3e(1 - Ūz*), (5)
k=1
k=1
s.t. Vj E [m], ون = [y, ... U5gl E 1),
Vk [g), au 0, 3x 0,
where dg and 8g are class-aware regularized parameters to control how many positive and negative labels would be included into model training for class k. Below, we primarily provide a solution of the optimization problem Eq. (5), and then discuss how set parameters ag and B& to capture the true class distribution of unlabeled examples.
Alternative Search It hard to directly solve the optimization problem 5g.55. since there are two sets of variables. feasible solution to adopt the alternative convex search 54 strategy that optimizes group of variables by fixing the other group of variables.

---

0.5
Estimatedd
Poporrae
Trao
0.3
0.2
0.0
Confidanca
20
40
60
70
80
Class Index
COCO
Prpprtzee
0.3
Confidence
0.2
Top-1
Class distrlaution-awarr thresholding
0.1
Top-
0.0
0
6
8
10
12
14
16
Instance aware thresholding
Class Indox
VOC
(a) Different Pseudo-Labling Strategies
(b) Class Proportions
Figure (a) An illustration of the comparison between instance aware and class-aware pseudo- labeling methods. (b) The curves of the estimated and true class proportions on COCO and VOC. By using the CAT strategy, CAP can provide high-quality pseudo-labels by approximating the true class distribution. This can be validate by the results in (b), where the empirical and true class proportions of positive labels show high-level consistency.
Suppose that pseudo labels ý are given, then the optimization problem Eq. can be transformed into ordinary loss by treating the pseudo labels the true ones:
min 72
1
((f(x))) v.)
m Σ (u(f(z;).
(6)
which can be solved by applying the stochastic gradient decent (SGD) method.
With the parameters 0 fixed, we reformulate the optimization problem with respect to ÿ as
min (nd((f((EE)) + (x)
agdi + 3x(1 - 998).
Consider that UN is assume to be one or zero, we can obtain the following solution:
if fx()) r(ou);
0
if (a S <(()), otherwise,
(8)
where 7(4)) = exp(-ax. and r(8g) exp(-3x) are two class-aware thresholds, and ÿN means that the label 0ı would not be used for model training.
3.3 Clas--iisribbution--aare Thresholding
An important problem is how the thresholds 7(ou) and T(x)), which determine the numbers of positive and negative pseudo-labels for every class k. In order to capture the true class distribution, we propose the Class-distribution-Aaaar Thresholding (CAT) strategy to determine T(0g) and 7(3x). Suppose that we are given yy, Vj e [m], f.e., the true label vectors of unlabeled training instances. By solving the following equation, can obtain ((a4) and 7(3)) that capture the true class distribution of unlabeled data.
Eee [T, 2 T(ak))
Σ I(fa(z;) S ((x))) = Pk:
7k
m
E Kıy=1)) and P mm [(pj==0) are respectively the proportions of positive and negative labels unlabeled data for class k. Although during the training process, the true labels of unlabeled instances are inaccessible, our observation shows that the estimated class distribution. i.c.,
where 7/
m

---

the class proportions of positive and negative labels in labeled examples, can tightly approximate the true class distribution. As shown Figure[[(b), we illustrate the proportions of positive labels in labeled examples and unlabeled examples for every class & on two benchmark datasets COCO and VOC. The proportions of labeled examples are respectively p = 0.05 and p = 0.1 for COCO and VOC. From the figures, it can be observed that even with small proportion of labeled examples (p = 0.05), it achieves a nearly complete overlap between the estimated and true curves, which validates that the estimated class distribution can be tight approximation of the true one. This motivate us to alternatively utilize the estimated class distribution to solve the solutions for T T(as) and 7 (Bkk):
Σ [(fa((()))))))))))))))))))))()) T(au)) 7k.
ET ((f((t))) 5T ((x)))
Pk.
- [yu= and Å& =0)= are respectively the proportions of positive and
where 2t
negative labels in labeled data for class k. Figure I provides an illustration of the comparison between three mastance-wwar pseudo-labeling methods and the CAP method. By utilizing CAT strategy, CAP is expected to assign pseudo-labed with the class distribution that approximates the true one.
In practice, to further improve the performance of CAP, one feasible solution is to discard fraction of unreliable pseudo-labels with relatively low confidences, which may have a negative impact on the model training. Specifically, for any class k E [g], we select top 7) 7 and roo f& proportion probable pseudo-labels, where 7), 1/0 E [0, I] are two parameters to control the reliable intervals
of pseudo-labels. By substituting the two terms into the right sides of (9. we can obtain the
thresholds correspondingly. In Section we perform ablation experiments to study the influence of reliable intervals on the model performance.
Theoretical Analysis
In this section, we perform theoretical analyses for the proposed method. In general, the performance of pseudo-labeling depends mainly on two factors, i.e., the quality of the model predictions and the correctness of estimated class distribution. Our work focuses on the latter. Consider an extreme case, where the model predictions are perfect, i.e., the confidences of positive labels are always greater than that of negative labels. such case, we still need an appropriate threshold to precisely separate the positive and negative labels. This implies that we need to capture the true class distribution of unlabeled data in order to achieve desirable pseudo-labeling performance.
4.1 Correctness of the Estimated Class Distribution
To study the correctness of estimated class distribution, we provide the following theorem, which gives an upper bound on the difference between the estimated class proportion 2 and the true class proportion 7 (its proof is given in Appendix A). A similar result can be derived for Pi-
Theorem 1. Assume the estimated class proportion f/k = i Tyya = 1), and the true class proportion > = ΣØ= I[yjk = for any l E [q]. where n and m are the numbers of labeled and unlabeled examples that satisfy m >> n. Then, with the probability larger than 2n 2m 2m we have, Vk E [g), 1iv 721 < ylog ylogm
22n
22m
Theorem I tells us that the correctness of the estimated class distribution mainly depends on the number of labeled and unlabeled data. In general, the bound dominated by the first term, since always satisfies m >> n. By neglecting the second term, we can see that Vk E [q], 7 + 4% in the parametric rate (I/Vn), where Op denotes the order in probability. Obviously, as the number of training examples increase, the estimated class distribution would quickly converge to the true one.
4.2 Generalization Bound
Moreover, we study the generalization performance of CAP. Before providing the main results, we first define the true risk with respect to the classification model f(æ; 0):
R(f) E(z.)) [f(f(x),)))

---

Our goal is to learn a good classification model by minimizing the empirical risk R(f) - R[(f)+ (f), where (f) and (f) are respectively the empirical risk of the labeled loss ()() and unlabeled loss (f(()) y):
R&(f) = 72 En y:),
- Co(f(z)), yz) m j=1
Note that during the training, we cannot train a model directly by optimizing Reg (f), since the labels of unlabeled data are inaccessible. Instead, we train the model with Rg) Σラ= Lu (f(z)) 9,))
where J, represents the pseudo-label vector of the instance æ.
Let (x)( y (f(())( (1 yt (o (x)) be the loss for the class k. and Le be any (not necessarily the best) Lipschitz constant of f. Let R. (F) be the expected Rademacher complexity 31 of F with N m + training points. Let = arg minjer R(f) be the empirical risk minimizer, where F a function class, and f = arg minfer R(l) the true minimizer. We derive the following theorem, which provides generalization error bound for the proposed method (its proof is given in Appendix B).
Theorem 2. Suppose that E(-) bounded by B. For some e if E [(fa(zy) 2 =(ax))
Iyyz 1)1/m S e for any & E for any O, with probability at least - 6, we have
log A 2N
R(s- - < (g + Bg
From Theorem 4 can be observed that the generalization performance of ff mainly depends on two factors, i.e., the pseudo-labeling егтоr and the number of training examples N. Apparently. a smaller pseudo-labeling error often leads to better generalization performance. Thanks to its ability to capture the true class distribution, CAP can achieve a much smaller pseudo-labeling error than existing instance-aware pseudo-labeling methods, which is beneficial for obtaining better classification performance. This can be further validated by our empirical results in Section 5.3 The second factor is the number of training examples. As N + oo and e + O, Theorem shows that the empirical risk minimizer f converges to the true risk minimizer f
5
Experiments
In this section, we first perform experiments to validate the effectiveness of the proposed method; then, we perform ablation studies to analyze the mechanism behind CAP.
5.1 Experimental Settings
Datasets To evaluate the performance of the propose method, we conduct experiments on three benchmark image datasets, including Pascal VOC-2012 (VOC for short) 2 [12), MS-COCO-200 (MS-COCO for short) (25,, and NUS- WIDE (NUS for short) 77. The detailed information of these datasets can be found in the appendix. For each dataset, we randomly sample a proportion p E {0.05, 0.1, 0.15, 0.2] of examples with full labels while the others without any supervised information. Following the previous works g 53 we report the mean average precision (mAP) on the test set for each method.
Comparing methods To validate the effectiveness of the proposed method, we compare it with five groups of methods: 1) three instance-aware pseudo-labeling methods: Top-1 (Eq.(2), Top-k (Eq. 3)) IAT (E),44); 2) two state-of-the-ar MLML methods: LL 199 (includes three variants LL-R, R. LL-Ct, and LL-CC), PI.C [46]; 3) Two sfate-of-the-are SSL methods: Adsh 14], Frec 4) One state-of-the-art SSMLL method: DRMI [4]] ; 5) Two baseline methods, BCF, ASL
DRMI. the only deep SSMLL method whose source code could be found on the Internet.
http://host.robott ac uk/ httpascca/VVC///////
attps://oocodaraset.org
htw.://wwww.....o..oo...ooooo/pa WIDE.
nus, ndw.p
htm

---

Table 1: Comparison results VOC and COCO in terms of mAP (%). highlighted in bold.
The best performance is
VOC
COCO
Method 0.05
0.1
0.15
0.2
0.05
p 0.1
p 0.15
0.2
BCE 67.95
75.35
78.19
79.38
58.90
63.75
65.91
67.33
ASL
71.46
78.00
79.69
80.77
59.12
63.82
66.10
67.51
L.-R
75.69
80.96
82.31
83.55
59.31
64.25
66.61
68.01
LL-Ct
75.77
81.04
82.31
83.50
59.33
64.23
66.69
68.11
LL-Cp
75.79
81.03
82.36
83.68
59.27
64.19
66.68
68.12
PLC
74.49
80.35
82.35
83.39
59.85
65.03
67.62
69.14
Top-1
75.77
80.78
82.65
83.72
57.62
62.84
65.50
66.96
Top-k
75.07
80.20
81.99
83,16
58.25
63.52
66.11
67.49
IAT
73.24
80.27
82.39
83,55
60.34
65.54
67.88
69.25
ADSH
75.37
80.34
82,80
83,93
60.75
65.37
67.70
69.01
FreeMatch
75.11
80.66
82.63
83,60
59.94
64.46
66.79
68.04
DRML
61.77
71.01
72.98
74.49
53.60
57.06
58.53
59.24
Ours
76.16
82.16
83.48
84.41
62.43
67.36
69.11
70.41
Table 2: Comparison results on NUS in terms of mAP (%).
The best performance is highlighted in
bold.
Method
ASL
LL-R
PLC
Top-1
Top-k
IAT
ADSH
FreeMatch
DRML
Ours
P 0.05
42.87
40.20
43.55
40.99
40.89
42.58
43,94
43.12
30.61
44.82
p
46.50
44.95
47.51
45.07
45.04
46.60
47.28
46.65
35.09
48.24
P 0.15
48.42
47.32
49.75
47.43
47.22
48.76
49.22
48.74
37.91
49.90
P 0.20
49.65
48.31
50.71
48.49
48.37
49.62
49.93
49.59
39.98
51.06
Furthermore, most MLML methods cannot be applied the SSMLL scenario, since they assume that a subset of labels have been annotated for each training instance. The detailed information of these methods can be found in the appendix.
Implementation We employ ResNet-50 [6 pre-trained on ImageNet [36 for training the clas- sification model. We adopt RandAugment 9 and Cutout [T0] for data augmentation. We employ A (29] optimizer and one-cycle policy scheduler 10 to train the model with maximal learning rate of 0.0001. The number of warm-up epochs is 64, and 64 for VOC, MS-COCO, and NUS. Furthermore, we perform exponential moving average (EMA) for the model parameter with decay of 0.9997. For all methods, use the ASL loss as the base loss function, since it shows superiority to BCE loss (34]. We perform all experiments on GeForce RTX 3090 GPUs. The random seed is set for all experiments.
12 for all datasets. The batch size is set 32,
5.2 Comparison Results
Table T and Table report the comparison results between CAP and the comparing methods in terms of mAP on VOC, COCO, and NUS. From the tables, we can see that: 1) DRML obtains unfavorable performance, even worse than basclines BCE and ASL, since performs two-stage training, which may destroy its representation learning. The original paper did not report the results on these three rather important to design an effective SSMLL method in deep learning paradigm. 2) CAP outperforms three instance-aware
datasets. Therefore,
sreuud-labbeing methods, which demonstrates that by utilizing CAT strategy, CAP can precisely estimate the class distribution of unlabeled data and thus obtain desirable pseudo-labeling performance. 3) The performance of CAP is better than that of two state-of-the SSL methods. achieve better performance, we have made several modifications for these two methods not limited to the following: stronger data augmentations; c) change the training scheme make them more suitable for the multi-label scenario. 4) CAP achieves the best performance in all cases and significantly outperforms
use the ASL loss b) adopt

---

CAP
Top-1
Top-k
IAT
70
76
CF1 Score (%)
65
Score (%9
14
72-
CF Score (9)
74
155
50
60
CH1
86
44.
68
70-
70-
CFI Score
70
72
68
5
7
9
11
13
11 13
Epochs
Epochs
Epochs
Epochs
60
CF1 Score (%)
65
CFI Score (%)
CF1 Score (%)
65
70
.55
CFI Score
50
53
53
45
50-
30-
50
40
45
45
45
5 7
5
15
13
Epochs
Epochs
Epochs
Epachs
(a) p 0.05
(b) p 0.1
p 0.15
(d) p 0.2
Figure 2: Pseudo-labeling performance in terms of CF1 score VOC, COCO. Each row corresponds
one dataset.
MAP Score (%)
MAP Score (16)
84
20
700
82
78
80
X2
30
MAP
4
66
68
MAN
64
68
74
82
66
0.80
0.85 0.90
0.95
1.00
0.95
0.96 0.97 0.98 0.99 1.00
0.80
0.85
0.90
0.95
1.00
0.95
0.96 0.97 0.98 0.99 1.00
no
7/1
no
(a) VOC, 77o =
(b) VOC, 1/1 =
(c) COCO, Yu
(d) COCO, m
Figure 3: Performance of CAP on VOC and COCO in terms of mAP (%) with the increase of mn and Vo.
the comparing methods, especially when the number of labeled examples small. These results convincingly validate the effectiveness of the proposed method.
5.3 Study on the Performance of Pseudo-Labeling
In this section, we explain why CAP better than the conventional instance-aware pseudo-labeling methods. Figure]jillssten iillustrates the performance of different pseudo-labeling methods in terms of CFI score on VOC and COCO. From the figures, we can see that CAP achieves the best performance in all cases. As discussed above, the pseudo-labeling performance mainly depends on two factors, i.e., the quality of model predictions and the correctness of estimated class proportions. CAP improves the pseudo-labelle performance by precisely estimating the class distribution. An interesting observation that at the first epoch, when the model predictions are the same for four methods, our method significantly outperforms the comparing methods, since it is able to capture the true class proportions. These results validate that CAP can achieve better pseudo-labeling performance.
5.4 Study on the Influence of Reliable Interval
As mentioned above, to improve the performance, instead of using all pseudo-labels, we can train the model with only reliable pseudo-labels within the reliable intervals that are controlled by 7h 7/-- Fig- ure illlustrates the performance of CAP as and 7/ change in the ranges of [0.8, 0.85, 0.9, 0.95, 1] and [0.95, 0.96, 0.97, 0.98, 0.99, 1]. From the figures, can be observed that discarding the unreli- able positive pseudo-labes would improve the performance, but discarding the unreliable negative pseudo-labels would degrade the performance. One possible reason behinds the phenomenon is due to the significant positive-negative imbalance in MILI data, i.e., the number of negative labels is often much greater than that of positive labels. This leads the model to be sensitive to false positive labels, while be robust to false negative labels. In our main experiments (Table I and Table 2 we set

---

= 1, = 1. In practice, we are expected to achieve better performance by tuning the parameter
?)
Conclusion
The paper studies the problem of semi-supervised multi-label learning, which aims to train multi- label classifier by leveraging the information of unlabeled data. Different from the conventional instance-aware pseudo-labeling methods, we propose to assign pseudo-labels unlabeled instances in a class-aware manner, with the aim of capturing the true class distribution of unlabeled data. Towards this goal, propose the CAT strategy to obtain estimated class distribution, which has been proven to be a desirable estimation of the true class distribution based on our observations. Theoretically, we first perform an analysis on the correctness of estimated class distribution; then, we provide the generalization error bound for CAP and show its dependence to the pseudo-labeling performance. Extensive experimental results on multiple benchmark datasets validate that CAP can achieve state-of-the ofrtthe-rrt performance.
In general, the performance of pscudo-labeling depends mainly two factors, i.c., the quality of the model predictions and the correctness the estimated class distribution. This work focuses on the latter. In future, we plan to boost the performance of SSMLL by improving the quality of model predictions.
Acknowledgments and Disclosure of Funding
Sheng-Jun Huang was supported by Natural Science Foundation of Jiangsu Province of China (BK20222012, BK2021 1517). the National Key R&D Program of China (2000AAA0107000), and NSFC (62222605). Masashi Sugiyama was supported by JST CREST Grant Number JPMJCR 18A2.
References
[1] Mokhtar Bazaraa, Hanif Sherali, and Chitharanjan M Shetty. Nonlinear programming: theory and algorithms. John Wiley & Sons, 2013.
[2] Emanuel Ben-Baruch, Tal Ridník, Itamar Friedman, Avi Ben-Cohen, Nadav Zamir, Asaf Noy, and Lihi Zelnik-Manor. Multi-label classification with partial annotations using class aware selective loss. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3764-4772 2022.
[3] David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi- superisiee learning with distribution alignment and augmentation anchoring. arXiv preprint arXiv: 191iv//911.00780 1.09785, 2019.
[4] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in neural information processing systems, 32, 2019.
[5] Gang Chen, Yangqiu Song, Fei Wang, and Changshui Zhang. Semi-supervised multi-label learning by solving sylvester equation. In Proceedings of the 2008 SIAM international conference on data mining, pages 410-419, SIAM, 2008
[6] Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo. Multi-lahel image recognition with graph convolutional networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 5177-5186, 2019.
[7] Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. Nus- wide: a real-world web image database from national university of singapore. In Proceedings of the ACM international conference on image and video retrieval, pages 1-9, 2009.
[8] Elijah Cole, Oisin Mac Aodha, Titouan Lorieul, Pietro Perona, Dan Morris, and Nebojsa Jojic. Multi-label learning from single positive labels. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 933-942, 2021.
10

---

[9] Ekin Dogus Cubuk, Barret Zoph. Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-TTien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems, 2020.
[10] Terrance Veeeies and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 04552, 2017.
[11] Thibaut Durand, Nazanin Mehrasa, and Greg Mori. Learning a deep convnet for multi-label classification with partial labels. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 647-657, 2019.
[12] Mark Everingham, SM Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111(():98-136, 2015,
[13] Vasilii Feofanov, Emilie Devijver, and Massih- Rezza Amini, Transductive bounds for the multi- class majority vote classifier. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 3566-3573, 2019.
[14] Lan Guo and Yu-Feng Li. iaaasaieblleeed semi-supervised learning with adaptive thresholding. In International Conference on Machine Learning, pages 8082-8094, 2022.
[15] Yuhong Guo and Dale Schuurmans. supersuper multi-label classification: simultaneous large-margin, subspace learning approach. Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2012, Bristol, UK. September 24-28, 2012. Proceedings, Purt 11 23, pages 355-370. Springer, 2012.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[17] Dat Huynh and Ehsan Elhamifar. Interactive multi-label enn learning with partial labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9423-9432, 2020.
[18] Jaehyung Kim, Youngbum Hur, Sejun Park, Eunho Yang, Sung Ju Hwang, and Jinwoo Shin. Dis- tribution aligning refinery of pseudo-label for imbalanced semi-supervised learning. Advances in neural information processing systems, 33:14567-14579, 2020.
[19) Youngwook Kim, Jae Myung Kim, Zeynep Akata, and Jungwoo Lee. Large loss matters in weakly supervised multi-label classification. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14156-14165, 2022.
[20] Xiangnan Kong, Michael K Ng, and Zhi-Hua Zhou. Transductive multilabel learning via label set propagation. IEEE Transactions on Knowledge and Data Engineering, 25(3):704-719 2011.
[21] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont- -ust,, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. International Journal of Computer Vision, 128(7):1956-199) 1981,, 2020.
[22] Jack Lanchantin, Tianlu Wang, Vicente Ordonez, and Yanjun Qi. General multi-label image classification with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16478-16488, 2021.
[23] Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, page 896, 2013.
[24] Junnan Li, Caiming Xiong, and Steven CH Hoi. Comatch: Semi-supervised learning with contrastive graph regularization. In Proceedings of the IEFF/CVE International Conference on Computer Vision, pages 9475-9484, 2021.

---

[25] Tsung- Yi Lin, Michael Maire, Serge Belongie. James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755, 2014.
[26] Shilong Liu, Lei Zhang, Xiao Yang, Hang Su, and Jun Zhu. Query2label: A simple transformer way multi-label classification. arXiv preprint arXiv:2107. 10834, 2021.
[27] Weiwei Liu, Haobo Wang, Xiaobo Shen, and Ivor W Tsang. The emerging trends of multi-label learning. IEEE transactions on pattern analysis and machine intelligence, 44(11):7955-79) 2021.
[28] Ziyi Liu, Zengmao Wang, and Bo Du. Multi-marginal contrastive learning for multi-label subcellular protein localization. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 20626-20635, June 2022.
[29] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, 2019.
[30] Gideon S Mann and Andrew McCallum. Simple, robust, scalable semi-supervised learning via expectation regularization. In Proceedings of the 24th international conference on Machine learning, pages 593-600, 2007.
[31] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018.
[32] Khoi Pham, Kushal Kafle, Zhe Lin, Zhihong Ding, Scott Cohen, Quan Tran, and Abhinav Shrivastava. Learning to predict visual attributes in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13018-13028, 2021.
[33] Kirill Prokofiev and Vladislav Sovrasov. Combining Metrie Learning and Attention Heads For Accurate and Efficient Multilabel Image Classification. arXiv e-prints, 2022.
[34] Tal Ridnik, Emanuel Ben-Baruch, Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, and Lihi Zelnik-Manor. Asymmetric loss for multi-label classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 82-91, 2021.
[35] Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertainty aware pseudo-label selection framework for semi-supervised learning. In International Conference on Learning Representations. 2020.
[36] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-222 2015.
[37] Wanli Shi, Victor S Sheng, Xiang Li, and Bin Gu. Semi-supervised multi-label learning from crowds via deep sequential generative model. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1141-1149, 2020.
[38] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raf- fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun BhannLLing Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-605 2020.
[39] Zixing Song, Ziqiao Meng, Yifei Zhang, and Irwin King. Semi-supervised multi-label learning for graph-structured data. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 1723-1733, 2021.
[40] Qiaoyu Tan, Yamming Yu, Guoxian Yu, and Jun Wang. Semi-supervised multi-label classification using incomplete label information. Neurocomputing, 260: 192-202, 2017.
[41] Thomas Verelst, Paul Rubenstein, Marcin Eichner, Tinne Tuytelaars, and Maxim Berman. Spatial consistency loss for training multi-label classifiers from single-label annotations. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3879-3889, 2023.
12

---

[42] Bo Wang, Zhuowen Tu, and John K Tsotsos. Dynamic label propagation for semi- supervised multi-class multi Muult classification. In Proceedings of the IEEE international conference on computer vision, pages 425-432, 2013..
[43] Lichen Wang, Yunyu Liu, Can Qin, Gan Sun, and Yun Fu. Dual relation semi-supervised multi- label learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 6227-6234, 2020.
[44] Yidong Wang. Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, Bernt Schiele, and Xing Xie. Freematch: Self- adaptive thresholding for semi-supervised learning. In The Eleventh International Conference on Learning Representations, 2023.
[45] Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, and Fan Yang. Crest: A class- rebalancing self-training framework for imbalanced remi-supervised learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10857-10866, 2021.
[46] Ming-Kun Xie, Jia-Hao Xiao, and Sheng-Jun Huang. Label-aware global consistency for multi-label learning with single positive labels. In Advances in Neural Information Processing Systems, 2022.
[47] Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi-supervised learning with dynamic thresholding. In International Conference Machine Learning, pages 11525-1 1536. PMLR, 2021.
[48] Yang Yang, Zano Fu, De-Chuan Zhan, Zhi- Bin Liu, and Yuan Jiang. Semi-supervised multi-modal multi-instance multi-label deep network with optimal transport. IEEE Transactions on Knowledge and Data Engineering. 33(2):696-777 709, 2019.
[49] Yang Yang. Yi-Feng Wu, De-Chuan Zhan, Zhi-Bin Liu, and Yuan Jiang. Complex object classification: multi-modal multi-instance multi-label deep network with optimal transport. In Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 2594-2603, 2018
[50] Wang Zhan and Min-Ling Zhang. Inductive semi-supervised multi-label learning with co- training. In Proceedings of the 23rd ACM SIGKDD international conference knowledge discovery and data mining, pages 1305-1314, 2017.
[51] Bowen Zhang, Yidong Wang. Wenxin Hou, Hao Wu, Jindong Wang. Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting sumirsspereieed learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems, 14411408--84111 2021.
[52] Feipeng Zhao and Yuhong Guo. Semi-supervised multi-label learning with incomplete labels. In Tounty-fourth international joint conference on artificial intelligence, 2015.
[53] Donghao Zhou, Pengfei Chen, Qiong Wang, Guangyong Chen, and Pheng-Ann Heng. Ac- knowledging the unknown for multi-label learning with single positive labels, 2022.
[54] Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized self-training. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pages 1982-5991,2011
13

---

A Proof of Theorem 1
Theorem 3. Assume the estimated class proportion 5n = 2. Myyi = 1), and the true class
proportion " m 2ー=1 [yjk = for any l. E [g], where Ti and m are the numbers of labeled and unlabeled examples that satisfy >> Then, with the probability larger than 2n- 2n we have, vk E [g), 176- 72l
y/og
y/og.m
v2n 22m
Proof. The proof is mainly based on Hoeffding's inequality that can be defined as follows.
Lemma 1. (Hoeffding's 's inequality). Let 21, .1,..., 2N be independent random variables bounded by (0s, bij. Then 2 = N Ei=1 Zi obeys for any v >> 0
2N
Pr(Ié - B 2 v) < 2exp(
Σ 1= (b; a,)²
Let 7/ = p(yk = 1) represents the expected class proportion. According to Hoeffding's inequality, for any 1: E [g). we have
Vlog /2n
Pr(ffx 7x =
2 2n
or equivalently, with the probability at least 2n -1 we have 1ix- 55 S vlogn/ nogn/V2n. /2n. Similarly,
with the probability least 2m for any [g]. we have h 5n S Vlog m/ m//.mmm.mmm /2m. By
applying the triangle inequality. with the probability least 1 2n 2m for any k [g]. have
Vlogn
Vlogm m
In -
v 22n
V2m
which completes the proof.
B Proof of Theorem 2
Theorem 4. Suppose that E(.) is bounded by B. For some > O. if Df [[a((zz)) 2 =((a))
Uμμj 1)1/m S for any k E for any 8 > O, with probability least - 6, we have
log 2 2N
R(Ò- - R(*) 2q 4gLe + BrB\
Proof. Before proving the theorem, we first provide two useful lemmas follows.
We primarily derive the uniform deviation bound between RF) and R()), which is a simple extension of the result in the binary setting 31
Lemma 2. Suppose that the loss function E is L Le-Lipschizz continuous w.E. 0. For any > 0, with probability least 8. we have
log
R(F) - -R)) 24LeRn+m + qBy 2(n m)
(1)
Proof. In order to prove this lemma, we define the Rademacher complexity of L and F with m training examples as follows:
Ru+m (L F)
E
sup
Σ
o;L(f(x;) 9:)++ + - ((,()((z)).)))
fEF
1=1
14

---

Considering that [(f(x), v) Ek= ((fk. yk). we have
Ratm S (R I {LeRnn+(FF)
(10)
where the second line is due to the Lipschitz continuity of the loss function &
Then, we proceed the proof by showing that the one direction sup R(f) R(f) is bounded with probability at least - 8/2, and the other direction can be proved similarly. Note that replacing an example (z,, My) with another (x',, vt)) leads to change of sup R(f) - R(f) at most _ due 4 to the fact that & is bounded by B. According to McDiarmid's ' inequality for any 5 > 0, with probability least 1 0/2, we have
jeyr
log 2
sup R() (())-R()) feF
R() S sup R() -R()) + +gB 2(n + m)
(11)
[Et
According to the result in [31 (Theorem 3.3) that shows EE R(f) R(f) 2 (F), by further considering the other direction sup R(f) - R(f), with probability at least - 6, we have
log 2 A
sup
R(f) A(f) s (m) qB
(12)
fer
2n + m
which completes the proof.
Then, we can bound the difference between R(f) and (f( as follows
Lemma
Suppose that E(-) is bounded by B. For some
0, if 1, 2 =(ax))
1y, = 1)l/m= S for any & E [g], for any f E F, we have:
RRn(S) (((D)-((())
qBe
(13)
Proof. Without loss of generality, assume that is the largest pseudo-labeling error among q classes, i.e. = maXke[g) E [xz))(z))) 2 r(ax)) - [[(y = 1)l/m. Obviously, consists of exactly two
types of pseudo-labeling error:
6 == (I) < <(()), Ush = =)
m
(14)
Σー I( 2 7 (ax Mik = 0)
60
m
where 6 calculates the proportion of positive labels being treated as negative ones, and co calculates the proportion of negative labels being treated as positive ones. Then, we prove the following two sides, which provide the bounds for R', (f). Firstly, prove its upper bound:
1 m
R(((F) m
ΣΣ
m
+ I(y = - -1o
m
+ +1((1 = O, f2(zz) 2 ((an))) + I(yjt = 1, < <(aw)))6((((z;;)
m
((f(z,).))) + ED [((f((z))) + Σ (o(fx(z,))
F=1
< (f) + qBe
15

---

where the second line holds based on 14. Then, we prove its low bound:
14
R&U) - 1
2 =(a
m 9
Iyyy +(y 0)(o(fa((;)
I(yy
m
I(yskk = 1,0(( < <T(x)))
I(Usk = 0, fa(z)) 2 zr(w)))((f((x;))
1
m
((((z)). Ub) - Σ (((b(1,))) - 60 Σ (o,)((r;)))
m
2 (f) gBe
By combining these two sides, we can obtain the following result:
RC(() R.)( - qBc
(15)
which concludes the proof.
For any 8 > O, with probability at least 1 - 6, we have:
R(F)
log 2
+ qB\ < Rf( + R. + (m) +qB
2N
log 2
< R((f) + + + (2) +
+ 9B1
2N
log 2
(16)
< R(() + R2,)) + gBe +2L
2N
log
< Rf) + Ra(f) 2qBe 2qL, qB
2.V
log 3z2
< R(J) 2qBe + 4gLaRx + B
2N
where the first and fifth lines are based on (1) and second and fourth lines are due to Lemma The third line is by the definition of f.
C Details of Experimental Settings
Table 3 reppors the detailed characteristics of four benchmark datasets. VOC is popular multi-label dataset that has been divided into a trainval set containing 5,011 examples and test set containing 4,952 examples from 20 object categories. MS-COCO is another widely used multi-label dataset, which consists of 82,081 training examples and 40,504 validation examples belonging to 80 different categories. In experiments, the validation set is used for testing. NUS is incomplete online due to many invalid URLs. Our collection consists of 126,034 training images and 84,226 testing images from 81 classes.
Comparing methods To validate the effectiveness of the proposed method, we compare it with four groups of methods: 1) Three instance-aware pscudo labeling methods: Top-1, which is similar to
16

---

Table 3: The detailed characteristics of benchmark datasets.
Dataset
Training
# Testing
Classes
VOC
5,011
4.952
20
COCO
82,081
16,416
80
NUS
126,034
84,226
FixMatch [8] that selects the most probable label as the ground-truth one; Top-k, which selects the top rank labels according to the predicted probabilities as the labels; IAT, which utilizes the instance-aware thresholding strategy to separates positive and negative labels. 2) Two state-of-the-art
MLML methods: LL [99 (includes three variants LL-R, LL-Ct, and LL-Cp), which rejects or corrects the large-loss examples to prevent model from memorizing noisy labels; PLC [46]. which performs the pseudo-labeling consistency regularization for recovering the labeling information of potential labels. 3) Two state-of-the- SSL methods: Adsh [14), which solves the flass-imblaanee issue in SSL by performing adaptive thresholding; FreeMatch [44], which improves the performance of FixMatch by introducing self-adas aelf-adaptive class fairness regularization. 4) One state-of state-of-the-art ttthhherrt
SSMLL method: DRML [43] which performs pseudo labeling by exploring the feature distribution and the label correlation simultaneously. 5) Two baseline methods, BCE, which is the most commonly used loss function for multi-label classification; ASL (34,, which improves BCE by utilizing the dynamic Nown-weighting and hard-thresholding techniques. DRML is the only deep SSMLL method whose source code could be found on the Internet. Furthermore, most MLML methods cannot be applied the SSMLL scenario, since they assume that subset of labels have been annotated for each training instance.
17


Reviews:
[{"review_id": "PQ1RVOIoDu", "reviewer": "Reviewer_ekLv", "summary": "This papers introduces a novel method called Class-Aware Pseudo-Labeling (CAP) to address the challenges of semi-supervised multi-label learning (SSMLL). Traditional pseudo-labeling methods struggle with instances associated with multiple labels and an unknown label count, often leading to the introduction of false positive labels or the omission of true positive ones. The CAP method overcomes these issues by performing pseudo-labeling in a class-aware manner. It introduces a regularized learning framework that incorporates class-aware thresholds, effectively controlling the assignment of positive and negative pseudo-labels for each class. The paper highlights that even with a small proportion of labeled examples, the estimated class distribution can serve as a reliable approximation. This observation led to the development of a class-distribution-aware thresholding strategy to align the pseudo-label distribution with the true distribution. The paper provides theoretical verification of the correctness of the estimated class distribution and offers a generalization error bound for the proposed method. Extensive experiments on multiple benchmark datasets confirm the effectiveness of the CAP method in addressing the challenges of SSMLL problems.", "questions": "In line 167 of the paper, the paper mention the use of an exponential transformation on the threshold value for each class in the Class-Aware Pseudo-Labeling (CAP) method. Why not just use a number as the threshold? Could the authors provide more details on the rationale behind this choice?", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The Class-Aware Pseudo-Labeling (CAP) method presents a significant innovation in the field of semi-supervised multi-label learning. Its unique approach of performing pseudo-labeling in a class-aware manner addresses the limitations of traditional pseudo-labeling methods, which often struggle with instances associated with multiple labels and an unknown label count.\n2. The authors provide a theoretical verification of the correctness of the estimated class distribution, which is a crucial aspect of the Class-Aware Pseudo-Labeling (CAP) method. This theoretical grounding not only validates the method's approach but also enhances its credibility and reliability. Furthermore, the authors offer a generalization error bound for the proposed method. This is an important contribution as it quantifies the expected performance of the CAP method and provides a measure of its robustness.\n3. The performance of the Class-Aware Pseudo-Labeling (CAP) method across all datasets and settings in the experiments is a strength of this paper. The fact that the proposed method consistently achieves optimal results demonstrates its effectiveness and robustness.", "weaknesses": "The paper does not provide a clear explanation for the choice of using an exponential transformation in line 167 on the threshold value for each class in the Class-Aware Pseudo-Labeling (CAP) method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your appreciation of our paper. We are glad that you considered our work “novel, a significant innovation, an important contribution”. We are glad to answer all your questions.\n\n**Q1:** The paper does not provide a clear explanation for the choice of using an exponential transformation in line 167 on the threshold value for each class in the Class-Aware Pseudo-Labeling (CAP) method.\n\n**A1:**  Considering the BCE loss ($\\ell_1(f_k)=-\\log(f_k)$ and $\\ell_0(f_k)=-\\log(1-f_k)$), for solving Eq. (7), according to [1], we obtain the closed-form solution  $\\hat{y}_k=1$ if $-\\log(f_k(x_i))\\leq\\alpha_k$; $(1-\\hat{y}_k)=1$, if $-\\log(1-f_k(x_i))\\leq\\beta_k$. With simple computations, we have $\\hat{y}_k=1$ if $f_k\\geq\\exp(-\\alpha_k)$; $\\hat{y}_k=0$, if $f_k\\leq 1-\\exp(-\\beta_k)$. For notational simplicity, we denote as $\\tau(\\alpha_k)=\\exp(-\\alpha_k)$ and $\\tau(\\beta_k)=1-\\exp(-\\beta_k)$. In the submitted paper, we mistakenly wrote as $\\tau(\\beta_k)=\\exp(-\\beta_k)$. We will correct this writing error in the revised version.\n\n[1] Self-Paced Learning for Latent Variable Models. NeurIPS 2010."}, {"review_id": "93zHEb8du8", "reviewer": "Reviewer_xWdd", "summary": "This paper proposes a Class-Aware Pseudo-labeling (CAP) method to solve semi-supervised multi-label learning (SSMLL) problem by controlling the assignment of positive and negative pseudo-labels for each class through a class-distribution-aware thresholding (CAT) strategy. ", "questions": "Please refer to the Weaknesses. ", "limitations": "The limitations of this paper should be discussed. ", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a pseudo-labeling strategy for semi-supervised multi-label learning in a class-aware manner.\n\n2. This paper is easy to read and well organized.\n\n3. The experimental results and the proof of Theorems in this paper seem to be solid.\n", "weaknesses": "1. Lack of some references. This paper is devoted to solving the SSMLL problem. Therefore, the authors should review and compare some SSMLL methods and more SSL methods. For example, \n[1] Rizve et al., In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning, ICLR, 2021. \n[2] Xu et al., Dash: Semi-supervised learning with dynamic thresholding, ICML, 2021.\n[3] Zhang et al., Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling, NeurIPS, 2021. \n\n2. What is the motivation for designing the operation $\\tau(\\alpha_{k})=exp(-\\alpha_{k})$? This does not seem to be mentioned in this paper.\n\n3. It is not enough clear about the Class-Distribution-Aware Thresholding. Also, it is suggested that the author gives a complete methodological illustration about CAP and CAT for each class to facilitate the readers' understanding. \n\n4. The random seed in this paper is set to 1 for all experiments. Following the SSL setting, it is necessary to report the mean value with different random seeds as the final performance.\n\n5. The authors should check if the title of the figure corresponds to the content. For example, Figure 2 is titled OF1 score while the results are CF1 scores. In addition, it is suggested to give a number for each formula.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your great efforts for reviewing our paper. We are glad that you considered our work “easy to read, well-organized, solid”. We are glad to answer all your questions.\n\n**Q1:** Lack of some references. This paper is devoted to solving the SSMLL problem. Therefore, the authors should review and compare some SSMLL methods and more SSL methods. \n\n**A1:** In our paper, we have compared the proposed method with the only SSMLL method found, DRML, and two recent SSL methods, Adsh (2022) and FreeMatch (2023), as well as multiple MLML (Multi-label Learning with Missing Labels) methods. We will add more SSL competitors in the future version. Also, we will improve the Related Works section according to the suggestions and add the mentioned relevant works to the revised version as follows.\n\nThe relevant works [1-3] will be added to the third paragraph of the Related Works section:\n\nTo improve the reliability of pseudo-labels, an uncertainty-aware pseudo-labeling method proposed in [1] selected reliable pseudo-labels based on the prediction uncertainty. \n\nUnlike FixMatch that selects unlabeled examples with a fixed threshold,  Dash [2] selected unlabeled examples with a dynamic threshold, with the goal of achieving better pseudo-labeling performance.\n\nFlexMatch [3] was proposed to select unlabeled examples for every class according to the current learning status of the model. \n\n[1] In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning, ICLR, 2021.\n\n[2] Dash: Semi-supervised learning with dynamic thresholding, ICML, 2021. \n\n[3] Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling, NeurIPS, 2021.\n\n**Q2:** What is the motivation for designing the operation $\\tau(\\alpha_k)=\\exp(-\\alpha_k)$?\n\n**A2:** For solving Eq.(7), according to [1], we obtain the closed-form solution $\\hat{y}_k=1$ if $-\\log(f_k(x_i))\\leq\\alpha_k$, that is, $\\hat{y}_k=1$ if $f_k\\geq\\exp(-\\alpha_k)$.  For notational simplicity, we denote as $\\tau(\\alpha_k)=\\exp(-\\alpha_k)$. We will include the above discussion in the revised version.\n\n[1] Self-Paced Learning for Latent Variable Models. NeurIPS 2010.\n\n**Q3**: It is not enough clear about the Class-Distribution-Aware Thresholding. Also, it is suggested that the author gives a complete methodological illustration about CAP and CAT for each class to facilitate the readers' understanding.\n\n**A3:** Thank you for your suggestion. As illustrated in Figure 1(a), Class-aware Pseudo-labeling (CAP) is a pseudo-labeling framework that differs significantly from conventional pseudo-labeling (Instance-Aware Pseudo-labeling, IAP). IAP performs pseudo-labeling for every instance (row), while CAP performs pseudo-labeling for every class (column). This implies that IAP needs to know in advance the label count for every unlabeled instance,  while CAP only needs to know the class proportions of unlabeled data. For IAP, Figure 1(a) lists three existing pseudo-labeling methods (Top-1, Top-k, instance-aware thresholding). Unfortunately, these methods cannot capture the true label count precisely. For our CAP, we propose the class-distribution-aware thresholding, which aims to use the estimated class proportions of labeled data to approximate the true class proportions of unlabeled data. Figure 1(b) shows that even with a small  number of labeled data (the labeled proportion $p=0.05$), the estimated and true class proportions highly agree with each other. We will add more notes and re-organize the caption of Figure 1 to make the illustration clearer in the revised version.\n\n**Q4:** The random seed in this paper is set to 1 for all experiments.\n\n**A4:** For the sake of easy reproducibility, following most of works in multi-label scenarios like [1-3], we report experimental results with a fixed random seed. Such an experimental setting has been adopted by most of multi-label learning literature. To make a fair comparison, the random seed was set as 1 for every comparing method.\n\n[1] Multi-Label Image Recognition with Graph Convolutional Networks. CVPR'19.\n\n[2] Multi-Label Learning from Single Positive Labels. CVPR'21.\n\n[3] Asymmetric Loss for Multi-Label Classification. ICCV'21.\n\n**Q5:** The authors should check if the title of the figure corresponds to the content. Figure 2 is titled OF1 score while the results are CF1 scores. In addition, it is suggested to give a number for each formula.\n\n**A5:** Thank you for your suggestions. The title was mistakenly wrote as OF1.  We will correct the title and label the formula with numbers in the revised version. \n\n**Q6:** The limitations of this paper should be discussed.\n\n**A6:** In general, the performance of pseudo-labeling depends mainly on two factors, i.e., the quality of the model predictions and the correctness of the estimated class distribution. Our work focuses on the latter. It is a promising future direction to boost the pseudo-labeling performance by improving the quality of the model predictions. A straightforward method is to use dual networks, which can prevent errors from accumulating on a single network, thus achieving better generalization performance. We will include a more detailed discussion in the revised version."}, {"review_id": "nsn0fKQyg5", "reviewer": "Reviewer_tMT4", "summary": "This work studies pseudo-labeling for multi-label semi-supervised learning. Differing from the traditional instance-aware pseudo-labeling methods, they propose to assign pseudo-labels to unlabeled data in a class-aware manner to capture the true class distribution of the unlabeled data. This work proposes CAT strategy to estimate the class distribution. It proves it is a desirable estimation by performing an analysis of the correctness of the estimation and providing the generalization error bound for CAP. ", "questions": "1. Adsh and FreeMatch are designed for traditional SSL, how do the authors implement these two algorithms for the SSMLL setting? \n2. This method can be used widely. I would re-consider rating if the authors are willing to make the codebase public. ", "limitations": "They do not claim/discuss any limitation in their main manuscript. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well-structured and easy to follow. The methods are clearly presented in Section 3. They also provide a theoretical analysis in Section 4 of the proposed method. Solid experiments are conducted to show the effectiveness of the proposed methods. Multi-label classification is an important task, and pseudo-labeling for this scenario will be helpful to this community. ", "weaknesses": "* The key idea relies on the observation that the class proportions of positive and negative labels in labeled examples can tightly approximate the true class distribution. I am concerned if the unlabeled data have totally different distribution, will the assumption relying on such observation still hold true? Therefore, it would be helpful to show some examples when there is a distribution mismatch.  \n\n* The technical novelty seems not that clear. It would be great if the authors can re-state and present the novelty of the propose approach. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments. We are glad that you considered our work “well-structured, solid experiments, helpful to the community”. We are glad to answer all your questions.\n\n**Q1:** The key idea relies on the observation that the class proportions of positive and negative labels in labeled examples can tightly approximate the true class distribution.\n\n**A1:** Many semi-supervised learning (SSL) methods have explicitly or implicitly adopted this assumption. For example, the early work [1] presented the expectation regularization, which encourages the the marginal distribution of model predictions on unlabeled data to match the marginal distribution of the ground-truth labels that was estimated during training based on the training examples. Many recent works [2-4] have applied this idea in deep semi-supervised learning. Compared with SSL that has achieved great advances,  semi-supervised multi-label learning (SSMLL) (in the context of deep learning) is still in its nascent period of development. Therefore, our paper focuses on the standard setting where labeled and unlabeled examples follow the same distribution.  \n\nAccording to your suggestion, to show the influence of the label shift on the final performance of our method, following the works [5-6], we draw $P(y)$  from a Dirichlet distribution with concentration $\\alpha$. We define the degree of label shift  $\\Delta=\\sum_{k=1}^q|\\hat\\gamma_k-\\gamma_k^*|$ between the labeled and unlabeled data. By using different values of $\\alpha$, the degree of label shift $\\Delta$ could be varied. From Table 1, we can see that the performance of CAP degrades slightly as the degree of label shift increases significantly. For example, when $p=0.1$ and $\\alpha=1$ (the degree of label shift has increased by ten times), the performance of CAP is still better than the comparing methods in Table 1 in the paper.\n\nWe will add a more detailed discussion of this point in the revised version. \n\nTable 1. MAP of CAP under different degrees of label shift. * denotes the result without introducing the label shift.\n\n| $\\alpha$  |   \\|   |   1   |  10   |  100  |     *     |   \\|   | $\\alpha$  |   1   |  10   |  100  |     *     |\n| :-------: | :----: | :---: | :---: | :---: | :-------: | :----: | :-------: | :---: | :---: | :---: | :-------: |\n| $\\Delta$  | **\\|** | 1.345 | 0.904 | 0.940 |   0.136   | **\\|** | $\\Delta$  | 1.428 | 0.998 | 1.019 |   0.098   |\n| $p$ = 0.1 | **\\|** | 66.17 | 66.86 | 66.74 | **67.36** | **\\|** | $p$ = 0.2 | 68.86 | 69.38 | 69.31 | **70.41** |\n\nRecently, several methods have been developed specifically to solve the label shift problem [5, 6], where the class marginal distributions of the labeled and unlabeled data are different. This is really an interesting future direction of our work. To solve the label shift  problem in the SSMLL scenario, a straightforward strategy is to estimate the class distribution of the unlabeled data using the well-established method [5, 6], and then apply CAP to obtain pseudo-labels.  \n\n[1] Simple, Robust, Scalable Semi-Supervised Learning via Expectation Regularization. ICML’07\n\n[2] ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring. ICLR’20\n\n[3] CoMatch: Semi-supervised Learning with Contrastive Graph Regularization. CVPR’20\n\n[4] AdaMatch: A Unified Approach to Semi-Supervised Learning and Domain Adaptation. ICLR’22\n\n[5] Detecting and Correcting for Label Shift with Black Box Predictors. ICML’18\n\n[6] LTF: A Label Transformation Framework for Correcting Target Shift. ICML’20\n\n**Q2:** The technical novelty seems not that clear. It would be great if the authors can re-state and present the novelty of the propose approach.\n\n**A2:**  Thank you for your suggestion. In SSMLL scenarios, conventional pseudo-labeling methods can hardly work when handling instances associated with multiple labels and an unknown label count. Our paper proposes a Class-Aware Pseudo-labeling (CAP) method to assign pseudo-labels in a class-aware manner, which is free of estimating the label count for each instance. This allows us to transform a hard problem, i.e., estimating the label count for each unlabeled instance, into a much easier problem, i.e., estimating the class distribution of unlabeled data. We further propose a class-distribution-aware thresholding strategy to reliably separate positive and negative pseudo-labels based on the estimated class proportions of labeled data, which tightly approximate the true class proportions.  We will re-organize the Introduction section based on the above discussion to highlight our technical novelty.\n\n**Q3:** Adsh and FreeMatch are designed for traditional SSL, how do the authors implement these two algorithms for the SSMLL setting?\n\n**A3:**  In our experiments, we compared our method with SSMLL methods and MLML (Multi-label Learning with Missing Labels) methods. To further validate the effectiveness of the proposed method, we also compared our method with the SSL methods additionally. As mentioned in line 283-284, to achieve better performance, we made several modifications for SSL methods: 1) use the ASL loss (also used in our method); 2) apply the same data augmentations as our method, including RandAugment and Cutout; 3) change the training strategy to make it more suitable for the multi-label scenario, specifically, employing AdamW optimizer and one-cycle policy scheduler. Furthermore, we also tuned the hyperparameters of these methods to achieve better performance.\n\n**Q4:** This method can be used widely. I would re-consider rating if the authors are willing to make the codebase public.\n\n**A4:**  We have submitted the source code of our method as supplementary material. We promise to make the code publicly available as soon as the paper is accepted.\n\n**Q5:** About the limitation of the paper.\n\n**A5:** Due to character limit, please refer to the beginning of the rebuttal."}, {"review_id": "YVfjsAhIiK", "reviewer": "Reviewer_wYDh", "summary": "The paper proposes a class-distribution-aware method to deal with the semi-supervised multi-label learning (SSMLL) problem. The main motivation is that the conventional pseudo-labeling methods cannot be applied to SSMLL scenarios, since instance is assigned with more than one ground-truth labels. To solve this challenge, the paper proposes a class-aware pseudo-labeling strategy, which is free of estimating the number of true labels for each instance. To capture the class distribution of unlabeled data, authors propose a regularized learning framework to determine the pseudo-labels by exploiting the estimated class distribution of labeled data. ", "questions": "The paper considers the estimation of class distribution of unlabeled data. Is there any other factor can affect the pseudo-labeling performance?", "limitations": "NA", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The motivation of this paper is clear. Although pseudo-labeling is a commonly used strategy in single-label case, it cannot be applied to multi-label case due to the unknown label count.\n\nThe proposed method is intuitive and effective. Authors propose the class-aware pseudo-labeling strategy to cleverly avoid the estimation of the label count for each instance. Consequently, the problem has been transformed to the task of estimating the class distribution of unlabeled data, which can be easily solved by the proposed class-distribution-aware thresholding method.\n\nTheoretical and empirical studies comprehensively verify the effectiveness of the proposed method. Specifically, the paper studies the correctness of the estimated class distributions theoretically and conduct comparisons with sota methods.\n", "weaknesses": "It seems that only one threshold can determine the positive and negative pseudo-labels for each class. It is suggested to explain the reasons why two thresholds are used in the paper.\n\nIt seems that the paper did not discuss how to set the parameter of ASL.\n\nThere are some language mistakes. The paper is suggested to proofread the paper carefully.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments. We are glad that you considered our work “well-structured, easy to follow”. We are glad to answer all your questions.\n\n**Q1:** It seems that only one threshold can determine the positive and negative pseudo-labels for each class. It is suggested to explain the reasons why two thresholds are used in the paper.\n\n**A1:** By utilizing two thresholds, our goal is to identify the positive and negative pseudo-labels with high confidences, which guarantees a high reliability of the pseudo-labels. By training on the reliable pseudo-labels, the model would achieve favorable performance.\n\n**Q2:** It seems that the paper did not discuss how to set the parameter of ASL.\n\n**A2:** Thank you for your suggestions. Since the ASL loss function is not our contribution, we followed the settings in the original paper to use the default values for all parameters. \n\n**Q3:** There are some language mistakes. The paper is suggested to proofread the paper carefully.\n\n**A3:** Thank you for you suggestions. We will carefully check the paper and correct the writing mistakes in the revised version.\n\n**Q4:** Is there any other factor can affect the pseudo-labeling performance?\n\n**A4:** In general, the performance of pseudo-labeling depends mainly on two factors, i.e., the quality of the model predictions and the correctness of the estimated class distribution. We will consider improving the quality of the model predictions in the future version."}]

Author rebuttal:
**About the limitation of the paper** (to reviewers xWdd, tMT4)

In general, the performance of pseudo-labeling depends mainly on two factors, i.e., the quality of the model predictions and the correctness of the estimated class distribution. Our work focuses on the latter. It is a promising future direction to boost the pseudo-labeling performance by improve the quality of the model predictions. A straightforward method is to use dual networks, which can prevent errors from accumulating on a single network, thus achieving better generalization performance. We will include a more detailed discussion in the revised version.

Meta-review: This paper studies the semi-supervised multi-label learning (SSMLL) problem. The authors propose a class-distribution-aware pseudo-labeling method termed CAP. Specifically, CAP adopts class-aware thresholds to improve the approximation of pseudo-labels, thereby improving the generalization effects. Both theoretical and empirical studies are conducted to demonstrate the proposed method.
The reviewers mostly hold positive ratings, except for some incomplete statements. During the discussion period, the authors provided more thorough explanations and results, and a reviewer increased the initial score. Overall, I recommend accepting this paper.
Decision: Accept (poster)