Semi-supervised Multi-label Learning with Balanced Binary Angular Margin Loss
Authors: Ximing Li
Silong Liang
Changchun Li
pengfei wang
Fangming Gu
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Semi-supervised Multi-label Learning
Variance Bias
Balanced Binary Angular Margin Loss
Self-training
Primary area: machine_learning_for_other_sciences_and_fields

Abstract: Semi-supervised multi-label learning (SSMLL) refers to inducing classifiers using a small number of samples with multiple labels and many unlabeled samples. The prevalent solution of SSMLL involves forming pseudo-labels for unlabeled samples and inducing classifiers using both labeled and pseudo-labeled samples in a self-training manner. Unfortunately, with the commonly used binary type of loss and negative sampling, we have empirically found that learning with labeled and pseudo-labeled samples can result in the variance bias problem between the feature distributions of positive and negative samples for each label. To alleviate this problem, we aim to balance the variance bias between positive and negative samples from the perspective of the feature angle distribution for each label. Specifically, we extend the traditional binary angular margin loss to a balanced extension with feature angle distribution transformations under the Gaussian assumption, where the distributions are iteratively updated during classifier training. We also suggest an efficient prototype-based negative sampling method to maintain high-quality negative samples for each label. With this insight, we propose a novel SSMLL method, namely Semi-Supervised Multi-Label Learning with Balanced Binary Angular Margin loss (S$^2$ML$^2$-BBAM). To evaluate the effectiveness of S$^2$ML$^2$-BBAM, we compare it with existing competitors on benchmark datasets. The experimental results validate that S$^2$ML$^2$-BBAM can achieve very competitive performance.

Semi-supervised Multi-label Learning with Balanced Binary Angular Margin Loss
Ximing Li
Silong Liang
Changchun Li Pengfei Wang
Fangming Gu
"oolleee of Computer Science and Technology, Jilin University, China
Key Laboratory of Symbolic Computation and Knowledge Engincering of Ministry of Education, Jilin University, China
CComputer Network Information Center, Chinese Academy of Sciences, China UUniversity of Chinese Academy of Sciences, Chinese Academy of Sciences, China liximing860gmail.. com, Baangchunli93@gnnn com, liangs123@mm 1130mails 1s jlu edu. pfwang@cnic. cn, gufmājuu. edu. cn
e...jju...uuc... cn,
Abstract
Semi-supervised multi-label learning (SSMLL) refers inducing classifiers using a small number of samples with multiple labels and many unlabeled samples. The prevalent solution of SSMLL involves forming pseudo-labels for unlabeled samples and inducing classifiers using both labeled and pscudo-labeled samples in a self-training manner. Unfortunately. with the commonly used binary type of loss and negative sampling, we have empirically found that learning with labeled and pseudo-labeled samples can result the variance bias problem between the feature distributions of positive and negative samples for each label. To alleviate this problem, we aim to balance the variance bias between positive and negative samples from the perspective of the feature angle distribution for each label. Specifically, we extend the traditional binary angular margin loss to a balanced extension with feature angle distribution transformations under the Gaussian assumption, where the distributions are iteratively updated during classifier training. We also suggest efficient prototype-basee negative sampling method to maintain high-quality negative samples for each label. With this insight, we propose novel SSMLL method, namely Semi- -uperviued Multi-Label Learning with Balanced Binary Angular Margin loss 2 -BAM).. To evaluate the effectiveness of S² BBAM, compare it with existing competitors benchmark datasets. The experimental results validate that MI, BBBM can achieve very competitive performance.
Introduction
Multi-label learning (MI.L) refers to the classification problem where each training sample can be associated with multiple labels [1. For example, in text categorization, a text can involve certain number of topics simultaneously
3: and in image annotation, an image can contain multiple objects of interest in one scene 4 5 Compared with single-label learning, MLL is more prevalent paradigm in real-world scenarios, and has been widely used in many applications such as
information retrieval 6 7 and recommendation systems [ D].
"Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Semi-supervised
Supervised
VDED 11
VDID
6.5 Eprcch
106
25
85
05
100
180
0.5
Epoch
Equch
Epucch
Figure 1: The variance difference hetween feature distributions (VDFD) of positive and negative sam- ples computed in semi-supervised and supervised manners across labels [6, 7, 14, 17) of VOC2012.
Despite the successful application of MLL, the competitive performance of most MLL methods heavily depends on the large volume of training samples with precise supervision 4 tunately, it is expensive to manually annotate each sample, so it is naturally time-consuming to collect loads of labeled training samples. Accordingly, the community has turned to alternative candidates to MLL,, and raised the question of whether one induce robust MLL classifiers with small number of labeled samples and a large number of unlabeled samples, which are cheaper to collect. This concept gives birth the emerging research topic of semi-supervised multi-label learning (SSMLL),
TO TT]. Unfor-
and many attempts have been recently proposed 12 13 15 16
Generally, the topic of SSMLL, as its name suggests, is in parallel inherited from semi-supervised learning (SSL) and MLL. The current prevalent ideas are estimating pseudo-labed of unlabeled samples with SSL techniques and inducing MLL classifiers with both labeled and pseudo-labeled samples in a self-training manner. Following the prior arts [18 19], the binary kind of losses, e.g.
binary cross-entropy loss and asymmetric loss (20). are commonly used to optimize MLL classifiers, where those are equivalent to optimizing the binary loss between the positive and negative samples for each label. To alleviate the imbalanced issue between positive and negative samples, especially for the scenarios with massive labels, the negative sampling tricks are often employed [21. 22 23) Unfortunately, in our preliminary experiments, we found such training paradigms suffer from the variance bias problem by using the labeled and pseudo-labeled samples in the context of SSMLL, since is difficult to guarantee estimating accurate pseudo-labels. To be specific, the problem implies that for each label, in SSMLL the variance difference between feature distributions of positive and negative samples is often larger than the ones in fully supervised learning, as illustrated in Fig [ In this situation, each trained binary boundary tends to keep away from the Bayesian optimal one, resulting in performance degradation.
To tackle this problem, we propose a novel SSMLL method, namely Semi-Supervised Multi-Labe Learning with Balanced Binary Angular Margin loss (BLML.-BBAM), The basic insight of 2L BBAM is to balance the variance bias between positive and negative samples from the perspective of the feature angle distribution for each label. To be specific, we extend the binary angular margin (BAM) loss, which measures the prediction loss by using the angle between the feature and binary boundary for each label. We suppose that for each label these feature angles of positive and negative samples are drawn from label-specific "positive" and "negative" Gaussian distributions. which are estimated by employing both labeled and pseudo-labeled samples during classifier training. Therefore, we can apply some linear Gaussian transformations over these feature angle distributions, so as to balance the variance bias between positive and negative samples for each label. Upon this idea, we design a new balanced binary angular margin (BBAM) loss and construct a novel 2 -BAM method based on the designed BBAM loss and self-training manner. We also suggest an efficient prototype-ber based negative sampling method to maintain high-quality negative samples for each label. We evaluate the proposed MI.. by comparing the most recent competitors on benchmark datasets. Experimental results indicate the superior performance of 22L22BBB
In summary, the main contributions this paper are listed as follows:
We develop novel SSMLL method, namely 2 2BBM, by balancing the variance bias between positive and negative samples from the perspective of the feature angle distribution for each label.

---

We design a new BBAM loss by extending the traditional binary angular margin loss with feature angle distribution transformations under the Gaussian assumption, and suggest an efficient prototype-based negative sampling method to maintain high-quality negative samples for each label.
We construct extensive experiments evaluate SBMI.2BBAM, and experimental results demonstrate the effectiveness of S ML2-BBAM.
2 Formulation and Analysis
2.1 Problem Formulation
By convention, we use x to denote the sample feature vector and y E {0.1]* the label indicator vector of K pre-defined classes, where 0/1 implies a sample irrelevan/releevant the category. In the task of SSMLL, we are formally given a collection of training samples DD {Di Du). where Di [(x]. y!) and D. {x]) >= are the collections of Ni labeled and N, unlabeled samples, respectively. The goal of SSMLL is induce classifier fw (x), parameterized by W, from D and use the classifier fw (x) to predict the label indicator vectors for future samples.
Broadly speaking, the classifier fw(x) (x) typically consists of backbone encoder and classification layer, parameterized by W and W respectively (i.e. W = { W))). Specifically, the backbone encoder transforms any original feature vector into more discriminative latent feature
fw (x);
the classification layer applies z to generate its corrsponding predictive logits p = fw (z). Given an SSMLL training dataset D. the classifier fw (x) is commonly optimized by minimizing the following generic self-traiming objective concerning W on Bi-sized labeled and sizei unlabeled batches:
1
Bi
K
A
B.
K
((W)
E(Pp Vir) +
(piko 9ik),
B,K
B,K
K=
where ((.,) .) a binary loss function; À is the coefficient parameter; p fw (xe) and fw (x,") are the predictive logits of labeled and unlabeled samples, respectively; y, is the pscudo-label of unlabeled samples induced from current classifier prediction p.".
2.2 How Variance Bias Affects the Performance
As shown in Fig we have observed that the generic self-training objective of SSMLL may suffer from the variance bias problem. Here, we discuss how it will affect the classification performance. We treat SSMLL as K independent semi-supervised binary classification (SSBC) tasks. For each SSBC task, let {(x,
y, U (x,} be the training data, where x RR and y* E [-1,+1} is the ground-truth label. Besides, let ÿ E {-1, +1} be the pscudo-label. For clarity and conciseness, we study the SSBC training data drawn from a mixture Gaussian distribution P which can be defined by the following distribution over (x, v) E Idd x {+1}:
+1, p a y
[W Σ& WW(- Σ
if +1; if = -1,
(2)
- 1, 11
where a is the prior probability of class "+1", μ = {MI,..., pa} T Σ = diag( (o
(d) }).
Σ_ diag( (a (1)
( "!! 2 with M 0. M + the distribution D* on the performance of the linear model fxx = sign((w, x) + b), where the
Hu, o(, of ( 0 vi E [d]. and (o
(a ( 1
.......""
We concentrate on analyzing the effect of the variance proportion M of
parameters w E IRd b E Ik, and sign(!) evaluates to +1 if scalar 1 2 0 and to -1 otherwise. For simplicity, denote
R(f, +1) E(x,))p (1(((x) - 1)ly= +1], R(f--1)
B(x))~ [1f(x) +1)1y -1],
where (t) is the indicator function that takes 1 where is true and O otherwise. We have the following theorems, whose proof can be found in the Appendix A
Theorem 2.1. Given an SSBC dataset with pseudo-labets S = (xx, ((x,, (i {(x,, 7.))
the optimal linear classifier feet minimizing the average standard classification error is given by:
3

---

fast arg mrg E(x,y)>s(1 (If (f(x) + y)].
(3)
When M > 1, it has the intra-class standard classification errors for the two classes
R(fsal: +1) (A M ((,,,
[[sast: -1) ( - M A+ v A2 q (M,,,,,,..)))
and when M- < 1, they are given by:
Rffaol: +1) + M (:M.::000AA(MMAAAMMMMMAAMA
R( - A VA2 ((M,,,e_,e)))
where (-)) } the cumulative distribution function (c.d.f.) of standard Gaussian distribution (0, 1).
A = (M²-1)E² ((,,o,c,,4)
2p
2log M C log a(2-c--26 22+4)
M1-1
(1-0)2-20--44)
ii=d Pi.
V Ei and {+_ are the proportions of negative instances being treated as positive ones and positive instances being treated as negative ones within pseudo-labes labels, respectively.
( = 1 i.e. M 1, the intra-class standard classification errors for the two classes can be expressed as follows:
R(fsst,+]) +1) ( 2p CY²
C23)
R(fsst, -1) $ -2p² + C2
(
2p
2p
Following 24 25 26 We employ variance of class-wise accuracy (VCA) to quantitatively measure
the model faimeess and present the definition of VCA below.
Definition 2.2. (VCA) Given classifier f X
where y 11, ,K). the variance
of wise accuracy of f is defined as VCA(f) = K Ex=(p(i)) - p). where p(() = P[f(x) = fly = = 1 1- + ily = and p 1 I p(i).
Theorem 2.3, Given an trained linear SSBC model feel in E3. the variance of class wise
accuracy VCA(fsal) increasing when M + for M > and M 0 for M < 1. Suppose 0, then when M = R(Sast +1) R(Jeat, 1) and VCA(Jea) = O.
log( [I-a) [1-a)[2-2c-
a12-r -2e,)
Remark 2.4. According to Theorem 2.3, the bigger or smaller value M will result in the increase of the variance of class-wise accuracy VCA(fast), which implies that the SSBC classifier fssl induced by (3. is unfair. Note that M is the variance proportion of feature distributions of positive and negative samples defined in (2). Therefore, improve the fairness of the induced classifier, propose to balance the variance bias of positive and negative samples for each label from the feature angle distribution perspective, leading our MLML
3
Proposed ML 2 -BAMM Method
In this section, we introduce the proposed SSMLL method named 2
3.1 Overview
Generally, our 2 built on the generic self self-training objective of SSMLL formulated by [ Specifically, propose novel Balanced Binary Angular Margin (BBAM) loss ( ))
aiming to balance the variance bias of positive and negative samples for each label from the feature angle distribution perspective with the Gaussian assumption. By applying our proposed BBAM loss to the generic SSMLL self-training objective in Eq the objective of MI...2BBA. can be formulated follows:
F
K
A
Ba
K
((W)=
Bul BuLRBBA(PLy M/r +
Ba @BRAM vi.),
(4)
B,K
B.
where
if (x;, )) E2i
Bix
1 if Ysk 1;
Ak E [K], Vi [Ni] [Nw).
otherwise,

---

and HR+ denotes high-quality negative sample sets constructed by negative sampling.
Here, pseudo-labels of unlabeled data {y" fi=1 i= are produced by employing the Class-Awer Pseudo- labeling (CAP) trick [16], which drives their label distribution towards the prior one that estimated with the labeled samples. Specifically, given the current classifier predictions {p.) No of unlabeled samples, {y Jiml i are given by:
pf >= òki
0 if Pt <= yAi otherwise,
4k E [K] vi (Na),
(S)
where the class-aware thresholds [= and {= are calculated by solving the equations:
>=6m) 1(
En 1(yh=1)
Nu
Vk E [K], Vi [Nu],
Dis 1(p <=m))
Ee
1(
and Vike -1 means that it will not be used for the classifier training.
3.2 BBAM loss
In this section, we introduce the proposed BBAM loss. As its name suggests, BBAM loss is extended from the Binary Angular Margin (BAM) loss, which measures the label-specific prediction risk by using the angle between the latent feature and boundary. Formally, for training sample yx), the BAM loss can be formulated as:
log( 1P m))
if if = 1;
(Pak BiR)
(6)
-log(1 10--a
if Yik = 0,
where Pia cos(Ou.) R. Will |1-112 is the f2-norm of vectors; z; and W denote the latent feature of sample and the weight vector of the classification layer for category k, respectively: 6s is the angle between 2; and W and m are the parameters used to control the rescaled norm and magnitude of cosine margin, respectively.
2, W
Reviewing the BAM loss in 6q one can observe that it calculates the loss by employing the label angles of samples for each category. We consider that trained binary boundary tends to deviate from the Bayesian optimal one for each category in SSMLL, where for most categories, the differences between feature distribution variances of corresponding positive and negative samples are much larger than ones in fully supervised learning. To address this issue, for each category k, suppose that label angles of its positive samples and ones of its negative samples are drawn from a label-specific "positive" Gaussian distribution N (( (x)) and label-specific "negative" one (( (")),)),, respectively. According to the properties of Gaussian distribution, we can easily transfer them into ones N (m(g), a and A (( (n) + with balanced variance कं (a])()) by performing the following Gaussian linear transformations on those label angles:
d (0sx) 0x 10) + b
(0xx) ag 0.s + +b
(p)
Ōk o (p)
(1)
(1 - (p) )
at
(n)
((n) = -11 (n) 1
(a)
Vk
[K]. (7)
o (a))
With these linear transformation pairs (v positive and negative samples can be refined into ones drawn from balanced angular distributions with one same variance, e.g.
(p) (-), vi (.)) for each category, label angles of both
(1)) 000 if yik 0. w/ (0x) N . if if = 1; (0w)))) ~N(u
5

---

Accordingly, the BAM loss in Eq 6 can be rewritten as the following BBAM loss:
log(
if yiu = 1;
(0sx))
Itc.
(Pa.
(8)
log(1 1+e 1+ (9 1)))
if Yik 0.
Estimating label angle variances. As mentioned above, concentrate on estimating label-specific "positive" and "negative" angular distributions, i.e. (N((L)(0)))))) (o)) and (0)) (o (o
for each category whose draws are the angles between its label proto- type C₁2 and latent features of its corresponding positive and negative samples, respectively. Here, we approximate { (2) labeled samples per-epoch.
((uk) (m) (of) ME and fcr
} k= with labeled and pseudo- k=1
For convenience, couple set of latent features and labels or pseudo-labels of training samples D in the current epoch. We calculate label prototypes {== } by averaging latent features of positive samples in
denote D (.)
(1) U ( y") the
No 1 = 1)z
C₁ =
N N.
1 = 1)
Vk E [K].
2==1
Consequently, the label angles between label prototypes and latent features of samples are given by:
Z: C2
Øsk = arccos(
) Ak E [K], Vi E [Ng + No),
|2
Accordingly, the estimations {(ug (o2) and (144)) (of)(0))) based on the current
negative sample sets {52x k= can be formulated as:
M 1 (Vik 1)0i
(of)(p) 2i-1 N 1(ya = 1) - 1x (1))2
(p)
Pk EM+NN 1 (Jiik 1)
EM 1( 1) -
E No Bax 1 = 0) (o.)(n)
E No Bx = () - 1/ (1))2
(m)
Mk Σ M+ B (yikk = 0)
Σn N 1 = 0) -
(10)
Besides, to avoid the misleading effect of false positive or negative samples, we also employ moving average with learning rate p over { (ol) (((b)) (of)())) and {c=
3.3 Negative Sampling
For efficiency, we suggest prototype-based negative sampling method. Specifically, for each label, tend select those negative samples that are more difficult to discriminate and would be more informative for the classifier training [21] [2). To achieve this, for each category, we measure similarity scores of negative samples based label prototypes [= and construct the nearest neighbor negative sample sets { 1-= as:
similar to its positive samples, because they
04 = ((x,, cx) E Rank(fd(z, cx) (x..))e)) (x) E Q Vk E [K].
where d(.) is the vector distance (e.g. cosine distance), Rank(.) outputs set of samples with the top-M minmum distance values; and ( is the negative sample set of category defined as:
a [(x), (x), y/) E Di, v = 0} { "x Du. vi, = 0}-
Accordingly, the final negative sample sets {A+ }== k= are generated by:
24 (xx, y: ) (x) ~ Uniform(S4.) (f+)} Vk E [K],
(11)
with size [[2,] = No k= where N = Σ proportion of positive and negative samples of each category. And we update those negative sample sets {2a k per-epoch for efficiency.
Di= (1// = 1) + E. (1 1), n controls the

---

Table Summary of the dataset statistics
Dataset
#Training
#Testing
#Classes
#Avg. Positive Classes
VOC
5,717
5,823
20
1.46
COCO
82,081
40.137
80
2.94
AWA
30,337
6,985
30.78
Ohsumed
22,054
10,300
23
1.65
AAPD
53,840
1,000
54
2.41
3.4 Model Training Summary
We describe the full training process MLL avoid inaccurate pseudo-labels in the early training stage, following [66]. warm up the classifier fw(.) (-) with the BAM loss of Eq 6 over labeled samples 2, by To epochs. Given the initialized fw(+), we continue to train it with the BBAM loss of Eq 8 over labeled samples Dj and unlabeled samples D., by T2 epochs. At each epoch,
kk=K {(ug (af) k==k we update pseudo labels {y: iy. by using Eq 5 label prototypes {cx JR=1
and {(1 (m)) by using Eqs and 10 and perform the negative sampling by using Eq
For clarity, the full training process is outlined in Appendix
4 Experiments
4.1 Experimental Settings
Datasets. We employ 5 widely used MLL datasets, including image datasets Pascal VOC-2012 (VOC) [27]. MS-COC02014 (COCO) [28) and Animals with Attributes2 (AWA) [2), text datasets Ohsumed 30 and AAPD 31. For clarity, the detailed characteristics of these datasets displayed in Table I Following [6 we transform these datasets into SSL versions. For each dataset, we randomly select training samples labeled ones, and the remaining unlabeled ones. We set π E (5%, 10%, 15%, 20%}. to explore the performance of our method under different data proportions. The image size resized 224 for all datasets.
Baselines. We employ 5 baseline methods for comparisons, including SoftMatch 32 FlatMatch 33 MIME [34), DRML [15], and CAP [16]. DRML and CAP are SSMLL methods; SoftMatch and FlatMatch are SSL. methods; MIME is single-positive multi-label learning (SPMLL) method. For SSL and SPMLL methods, we follow CAP to apply them to SSMLL tasks.
Evaluation metrics. We employ 5 evaluation metries, including Micro-F1, Macro-F1, mean average precision (mAP), Hamming Loss and One Loss [ and compute them with the Scikit-Learn tool
Implementation details. We use the pre- pre-trained ResNet-50 [35 the backbone for image datasets and SERT-base-uncaat size is 32 for VOC, 128 for AWA and 64 for COCO, Ohsumed and AAPD. The warm-up epoch To is 12. The 8 and m are 20 and 0.4 in VOC, 20 and 0.3 in COCO, 10 and 0.2 in AWA, Obsumed and AAPD. The parameters for negative sampling 17 set to 5.
model (36) for text datasets. We set the decay of EMA as 0.9997. The batch
4.2 Results
The experimental results are presented in Sable Table Overall, our method achieves good performance all metrics. Our model ranks 1st on average five datasets and has significant advantage over baselines. The detailed analyses are presented as follows.
We can observe that 3 has advantages over recent SSMLL methods. Especially in the Micro-F1 and Macro-F1, our method has significant improvement. On both VOC and COCO, our Fl and mAP values increase by average of 0. and 0.01. Furthermore, on Ohsumed and AAPD, we surprised to discover from the results that our method also has good results. In all data proportions, the average improvement the mAP is 0.11, 0. 0414 on Macro-F FFl and 0.19 on Micro-F1. This result is foreseeable because our method balanced angle variance using
Comparing with SSMLL methods:
hhtpss//scikit-eemmooo///ss

---

Table 2: Experimental results on images datasets. The best results are highlighted in boldface.
VOC
Methel
MicoFitt
Micr-PIf
RAPT
Lieeesin Lews)
Clac Lowe
1.6
102
100
135
T- 115
7- 175 7-
SEX
F- 755
Se0Maachh
DX662
0.7187
B.0611
003444
00620
0.931
0L5755
0,335
00233
0L7721
0.187
0.0019
0AM8
0.1655
0.146
0.1143
0.803
DRMB.
06690
0,655
07226
105535
0.5900
0.5W
DAMM
077199
0.005
113333
111312
1561
0C518
01007
0.338
0.770
01512
CAP
D6002
0.603
0.679
000073
[U9222
CADKS
04535
015335
0.7628
D/8216
018348
0.5460
0 08000
0L5535
0.0622
0.130
0.0011
0.621
BU755
8.7997
0 0.540
8.443
08845
6.7396
0,0015
0,1124
081141
0.7866
0,8345
0.6444
0 0.545
0.8308
0,328
0.604
8A233
9.1087
0.6667
0.887
00795
COCO
Micm-1tt
Macro-111
IRAM
Hureenng Lows,
Laua)
MIME:
026622
0.477
0-9AM
165122
012557
0.7751
0U404)
LL5372
1L5379
015633
0 1300
02500
0.1455
DRMEL
DX071
0,6226
0/692
06688
0,3345
0,50%
0L3779
0.5567
0.5188
0.5461
0.5026
0.6177
0 02242
06342
020000
5/225
0.1455
0.1285
0.1245
0.009
CAP
0.627
0.567
0.5724
[U523]
0.596
0L5422
015415
0.5243
160336
0.891
0.764
0.823
0C552
0.0400
51058
0.104
0.0041
B0726
0V000)
8,7074
8.7100
0.734
64141
8.600
0.694
0.6726
0.634
0.6886
0 7003
0 0030
0L8222
0.006
RAOM
0.100
0.661
0.004
00799
JWAA
Metrol
Macn-111
MAP
SafMtasch
0:000
6077
01334
0.776
0.579
0.553
0.115
0.3588
D6554
115191
0.1500
031555
02133
02176
15550
0.1991
0.19/
FlatMaich
DW015
0.6077
D/619)
003113
0U321
0.5-77
(L3907
013535
03323
06499
015555
0.6STT
01100
1L2367
02106
02166
0.112
010005
0.1116
0.1122
MTME
0.1770
0 13560
0.499
0.990
00705
0.180
0.0499
02327
0,9922
0,3033
0.4762
0.5365
0 03500
0.3900
0 3064
3,5012
0.1650
0.2061
0.106
0.2064
DRMEL
D4627
0 6856
0/942
06993
0.599
6.554
0,5727
0.5585
0.550
06926
05537
0.355
0 2355
0.2230
0.226
0.1360
0.1801
0.000
0.699
0/000
0.991
0.304
0.53
0.999
06615
015400
06651
3130
01093
0119
67213
0.7255
8:7215
01229
0.5833
0.3114
0.3933
0L3944
0.419
014633
(15478
018468
0.2892
(L2860
0.3100
9,342
0.126
0.310
0.1149
0.1181
Table 3: Experimental results on text datasets. The best results are highlighted in boldface.
Guamal
Metvol
Fit
IAPP
Rerering Tows)
Lons)
Se0Mathh
B:709
0:2306
0.4654
0.5006
0.5302
0.9766
0.225
0.5086
0,5074
0.514)
FlaMaidh
05161
0.4866
0/234
01442
0L3073
0.2222
0L1904
0L1775
0.137
04751
0.4993
0,5129
0.6699
0L5143
0.0631
30099
0.343
0.4415
0.5844
0,508
DRVB.
0.9955
0 2010
0:4155
0.905
0.193
0.1972
0.1955
0.000
0.133
0.991
0.3083
0.140
0 05599
0.668
0.063
3/851
0.000
0.567
0,5700
0.50M
CAP
0.062
0.5776
0.501
165455
0.115
0.5245
155334
114733
165770
IL5740
03505
01052
03257
03336
0251
B2WS
R.671
8.7100
8.719
IL7598
BA779
1L3128
0.5537
BUSSES
IL6684
4.884
0.8667
IL840R
0.117
0.3166
0.38
0.770
AAPD
Metvol
M1t
MAP
Binering
H0X
150
125
355
Sefhtaach
02345
0.3025
D00025
103274
DU3612
CUISE
DU3530
IL0483
IL3753
ILDAR
033550
ILC59S
DIMIC
(15630
DAAZ
FalMaida
03021
0.3147
03135
03155
00519
C,499
00437
0.443
03573
03306
0,3570
0,3621
0.0607
0,514
0.0613
0,660
0.5631
0.665
006344
DRVB.
D/160
0 41000
0/077
B10M
0.0015
01:555
B1 15757
(1 1595
0 02515
00576
0
55500
01591
0 5,500
CAP
0.5222
0 0.5726
8.500
0.935
0.9917
0.310
0.4257
0.4051
0.4355
0.9995
0.4899
0 0432
0.061
5.50
0.361
0.255
0.2544
8.7157
8.7279
8.3322
0.5971
8.503
0.5786
1L3823
0.5133
IL5894
0.3900
1.8262
IL8243
BJCWW
0.3321
0.1580
0.1500
8.19A)
Gaussian transformation, making the prediction of pseudo labels more accurate. Since Ohsumed and AAPD are text datasets, this result also demonstrates the good universality of our method.
Comparing with SSL methods: Our MI. 2- -BBAM improves in both F1 and mAp metrics. For example, at π = 5%, the mAP of S 2 is 0.07-0. 16 higher than SoftMatch and 0.05-0.14 higher than FlatMatch across all datasets. We believe that this is because both methods are applied to multi classification tasks. So during the training process, it is more inclined to make single label classification decisions. Therefore, it doesn't 't perform well as the SSMLL method. It can be inferred that it is important to set a dedicated method for SSMLL tasks.
Comparing with SPMLL methods: We observe that the performance of MI. --BM-BB is better than MIME in all aspects. When π = 5%, the average improvement on the mAP is 0.11, 0.42 on Macro-FI and 0.41 on Micro-F1. We believe that this is because SPMLL is primarily designed to address the issue of incomplete labels. However, there a large amount of unlabeled data in the setting of SSMLL tasks. This leads to the MIME method being unable to obtain single positive observation labels for these data, resulting in a significant loss of information. Therefore, the performance of MIME has declined.
8

---

Saiifaach
Flahlards
330
Spach
Epech
Figure 2: Comparison of VDFD on VOC2012.
Table 4: Results of the ablative study on VOC2012 and COCO.
7 10%
7 20%
S'stL" -BBAM
Wio BEAM
S'ME -BBAM
wio BBAM
5'ME -BIAM
We BBAM
S'ME -BBAX
Mil BBAM
0.7845
Macro-F11
0.7906
0.7247
0.8015
0.7799
0.8134
O.
0.8141 0.8458
07957
mAP
0,7866
0,7881
03345
0.8204
0.5454
0,8274
0.5252
0OC1)
S''m -BBMM
wis BBAM
S MIE
-BBAM
wio BBAM
S ME -DDAM
wio BIAM
S KIx.
-BBAN
w/o BBAM
Micro
0.6981
0.6952
0,7052
RAP
0.6354
0.5994
0.6741
0.6316
0.6896
0.6520
0,7023
0.6628
4.3 Ablation Study
To evaluate the effectiveness of the proposed BBAM loss, perform several ablative studies by replacing it with the BAM loss (denoted by "w/o BBAM") on VOC2012 and COCO. The results of the classification performance and VDFD present in Table 4 and Fig respectively. It clearly demonstrates that the proposed BBAM loss can significantly improve the classification performance and reduce variance differences between feature distributions. These results are expected because the BBAM loss can balance the variance bias between positive and negative samples from the perspective of the feature angle distribution for each label, leading to fairer MLL classifier. Besides, we can observe that the VDFD of our S ML -BBAM is much lower than those SSMLL baselines during the training procedure, further proving the effectiveness of the BBAM loss in balaneing the variance bias.
4.4 Parameter Evaluation
We conduct experiments on our method under different parameter settings. The experimental results shown in Fig We fix the m value to 0.4 and set the s values to [1,10, 10, 20, 30, 40, 50} respectively. When is set between and 10, the performance increases with the
Performanee
And when is set between 10 and 50, there is significant change in the performance. One possible reason for this situation is that when the is small, the convergence speed of the model is too slow. So by the end of training, the model is not yet its optimal state. We also explore the best accuracy by setting different cosine margins. We fix the value of to 20 and set the values of m to {0.0, 0.2, 0.4, 0.6, 0.8, 1.0] respectively. We find that the performance
Micro-
Macro-F1
Macro-F1
Figure 3: The sensitivity analysis of the rescaled norm and magnitude {s, m] of cosine margin on VOC2012 with œ = 5%.
its optimum at 0.4.
Related Works
SSMLL methods. Recently, SSMLL has received lot of attention. In the latest methods, most focus on how to use the connections between labels to guild training on unlabeled data. For instance, SMILE 37 calculates the relationships between labels by constructing adjacency graphs, while COIN 14 applies the well-known co-training method and learns under inductive settings. DRMI. 15
constructs the relationship network between labels by designing two classifiers and adopting

---

domain adaptation strategies. At the same time, how to effectively align pseudo labels with real labels also an important issue. CAP [66 developed class- distribution-awe -ware thresholding strategy to control the assignment of positive and negative pseudo-labels. However, the current SSMLL methods have not paid attention the variance bias problem, which affects the performance of the methods.
MLL methods. MLL has multiple research directions. Some methods focus on the model structure. For instance, 338 proposed a graph convolutional networks model improve the performance of multi-label image recognition. 4 proposes a unified framework that combines CNNs and RNNs.
Some others focus on exploiting label correlations improve performance. LSF-CI 39 calculates instance correlation in the feature space and label correlation in the label space through a probabilistic neighborhood graph model and cosine similarity. Due to the complete label information of the training samples, the MLL method can theoretically achieve Bayesian optimal classifier boundaries. However, in semi supervised learning, incorrect pseudo labels may provide incorrect guidance for classification boundaries.
SSL methods. Pseudo Label [40] is one of the earliest semi-supervised learning methods for neural networks. It generates pseudo labels for unlabeled data and continuously improves the accuracy of pseudo labels the model is optimized. As data augmentation technology has advanced, an increasing number of SSL methods are incorporating this technology 41 research has been conducted on the threshold issue of pseudo labels dynamic threshold strategies, they have been able to obtain more accurate pseudo labels, effectively enhancing the performance of the SSL methods. order utilize pseudo labels with low confidence but correct classification, 32 proposes an effective method that fits the confidence distribution of truncated Gaussian functions. Moreover, [33] discovered that the generalization ability of SSL. models is impacted by disconnection between labeled data and unlabeled data, and proposed the FlatMatch method to address this issue. However, it's important to note that these SSL methods are designed to handle multi-class single-label tasks [50 51 and cannot be directly applied to multi-label learning scenarios.
43
45
46). Further
49 By developing
Conclusion
In this paper, we proposed a novel SSMLL method, namely SiML2-BBAM. Our SLMLL-BBAN balances the variance bias between positive and negative samples from the perspective of the feature angle distribution for each label. To achieve this, we design novel balanced binary angular margin loss by extending the traditional binary angular margin loss with feature angle distribution transformations under the Gaussian assumption, where the distributions are iteratively updated during classifier training. We also suggest an efficient prototype-based negative sampling method maintain high-quality negative samples for each label, Empirical results demonstrate that our 2 -BBAM outperforms current SSMLL baseline methods.
Limitations
From the empirical results, we found that SiML,2BBAM suffers from slightly lower mAP scores on the benchmarks VOC and COOC when increasing the proportion of labeled training samples. This may restrict the range of applications and scenarios in which S² ML -BBAM can be effectively used. And we will further exploit in our future works.
Broader Impacts
The paper focuses solely on the technical aspects of SSMLL algorithms. Therefore, this work can benefit wide range of machine learning researchers. Also, we do not expect our efforts to have any negative consequences.
Acknowledgements
We would like to acknowledge support for this project from the National Science and Technology Major Project of China (No.20212D01 12500), the National Natural Science Foundation of China (No.62276113) and China Postdoctoral Science Foundation (No.2022M721321),
10

---

References
[1] Zhang, M., Z. Zhou. review multi-label learning algorithms. JEEE TKDE, 26(8):1819- 1837, 2014.
[2] Fujino, A., Isozaki, Suzuki. Multi-label text categorization with model combination based on fl-score maximization. In IJCNPP, pages 823-828. 2008.
[3] Wu, H., S. Qin, R. Nie, et al. Effective collaborative representation learning for multilabel text categorization. IEEE TNNLS, 33(10))5200-577) 5214, 2021.
[4] Wang, J., Y. Yang, J. Mao, et al. Cnn-rnn: unified framework for multi-label image classifica- tion. In CVPR, pages 2285-2294 2016.
[5] Lanchantin, J., Wang, V. Ordonez, et al. General multi-label image classification with transformers. In CVPR, pages 16478-16488. 2021.
[6] Zhao, F. Y. Huang, L. Wang, et al. Deep semantic ranking based hashing for multi-label image retrieval. In CVPR, pages 1556-1564. 2015.
[7] Lai, H., P. Yan, X. Shu, et al. Instance-aware hashing for multi-label image retrieval. IEEE TIP, 25(6):2469-222 2ete299999-29,, 2016.
[8] Zhang. D., S. Zhao, Z. Duan, et al. A multi-label classification method using hierarchical and transparent representation for paper-reviewer recommendation. ACM TOIS, 38(1):1-20, 2020.
[9] Izadi, M., A. Heydarnoori, G. Gousios. Topic recommendation for software repositories using labti classification algorithms. Empirical Software Engineering. 26(5):93, 2021.
[10] Zhu, F., H. Li, W. Ouyang, al. Learning spatial regularization with image-level supervisions for multi-label image classification. In CVPR, pages 5513-5522. 2017.
[11] Guo, H., Zheng, X. Fan,
Visual attention consistency under image transforms for multi-label image classification. In CVPR, pages 729-739. 2019.
[12] Wang. B., Z. Tu, J. K. Tsotsos. Dynamic label propagation for semi-supervised multi-class multi-label classification. ICCV, pages 425-432. 2013.
[13] Zhao, F. Y. Guo. Supi-supe multi-label learning with incomplete labels. In ISCAI, pages 4062-4068. 2015.
[14] Zhan, W., M. Zhang. Inductive semi-supervised multi-label learning with co-training. In SIGKDD, pages 1305-1314. 2017.
[15] Wang, L.. Y. Liu, C. Qin, et al. Dual relation semi-supervised multi-label learning, In AAAI, pages 6227-6234. 2020.
[16] Xie, M.-K., J.-H. Xiao, H.-Z. Liu, et al. Class-distrbuttion-awaar pseudo-labeling for semi- supervised lulti-label learning. In NeurIPS. 2023.
[17] Zaitian, W., W. Pengfei, L. Kunpeng, et al. A comprehensive survey on data augmentation. arXiv preprint arXiv:2405.0959), 2024.
[18) Cole, E., O. M. Aodha, T. Lorieul, et al. Multi-label learning from single positive labels. In CVPR, pages 933-942. 2021.
[19] Baruch, E. B., T. Ridnik, I. Friedman, al. Multi-label classification with partial annotations using class-aware selective loss. In CVPR, pages 4764-4772. 2022.
[20] Ridnik, T., E. Baruch, N. Zamir, et al. Asymmetric loss for multi-label classification. In ICCV, pages 82-91. 2021.
[21] Jiang, T., D. Wang, I.. Sun, et al. Lightxml: Transformer with dynamic negative sampling for high-performance extreme multi-label text classification. In AAAI, pages 7987-7994. 2021.
[22] Dahiya, K., D. Saini, A. Mittal, et al. Deepxml: A deep extreme multi-label learning framework applicd to short text documents. In WSDM, pages 31-39. 2021.
[23] Qaraci, M., R. Babbar. Meta-classifier free negative sampling for extreme multilabel classifica- tion. Machine Learning, pages 1-23, 2023.
[24] Ma, X., Z. Wang, W. Liu. On the tradeoff between robustness and fairness. In NeurlPS. 2022.
[25] Caton, S., C. Haas. Fairness in machine learning: survey. ACM Computing Surveys, (6(7):166:11166::: 2024.

---

[26] Mehrabi, N., F. Morstatter, N. Saxena, et al. A survey on bias and faimess in machine learning. ACM Computing Surveys, 14:6::115:11115:35, 2022.
[27]
Everingham, M., S. M. A. Eslami, L. V. Gool, et al. The pascal visual object classes challenge: A retrospective. 1JCV, (11(1))9--136) 2015.
[28] Lin, T., M. Maire, S. J. Belongie, et al. Microsoft coco: Common objects in context. In ECCV,
pages 740-755. 2014.
[29] Lampert, C. H., H. Nickisch, S. Harmeling, Atrrebute-based classification for zero-shot visual object categorization. IEEE TPAMI, 36(3):453-465. 2013.
[30] Moschitti, A., R. Basili. Complex linguistic features for text classification; A comprehensive study. In ECIR, pages 181-196. 2004.
[31] Yang. P.. X. Sun, W. Li. et al. Sgm: Sequence generation model for multi-label classification. In COLING, pages 3915-3926. 2018.
[32] Chen, H., R. Tao, Y. Fan, et al. Softmatch: Addressing the quantity-uuality trade-off in semi-supervised learning. ICLR, 2023.
[33] Huang, 7., L. Shen, J. Yu, et al. Flatmatch: Bridging labeled data and unlabeled data with cross-sharpness for semi-supervised learning. NeurIPS. 2023.
[34] Liu, B., Xu, Lv, et al. Revisiting pseudo-label for single-positive multi-label learning. In ICML, pages 22249-22280 2023.
[35] He, K., X. Zhang, S. Ren, et al. Deep residual learning for image recognition. In CVPR, pages 770-778. 2016.
[36] Devlin, J., M. Chang, K. Lee, al. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pages 4171-4186. 2019.
[37] Tan, Q. Y. Yu, G. Yu, et al. Semi-supervised multi-label classification using incomplete label information. Neurocomputing, 260: 192-202, 2017.
[38] Chen, Z., X. Wei, P. Wang. al. Multi-label image recognition with graph convolutional networks. CVPR, pages 5177-5186. 2019.
[39] Zhang, J., C. Li, D. Cao, et al. Multi-label learning with label-specific features by resolving label correlations. KBS, 159:148 157, 2018.
[40] Dong-Hyun, L., et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In ICML Workshop, page 896. 2013.
[41] Berthelot, D., N. Carlini, I. J. Goodfellow, et al. Mixmatch: A holistic approach to semi- supervised learning. In NeurIPS, pages 5050-5060. 2019.
[42] Zhang, H. M. Cissé, Y. N. Dauphin, et al. mixup: Beyond empirical risk minimization. In ICLR. 2018.
[43] Berthelot, D., N. Carlini, E. D. Cubuk, al. Remixmatch: Semi supervipe learning with distribution alignment and augmentation anchoring. ICLR, 2020.
[44] Sohn, K. D. Berthelot, N. Carlini, et al. Fixmatch: Simplifying semi supervise learning with consistency and confidence. In NeurlPS, pages 596-608 2020.
[45] Li, C., X. Li, L. Feng, al. Who your right mixup partner positive and unlabeled learning. In ICLR. 2022.
[46] Li, C., Y. Dai, L. Feng, et al. Positive and unlabeled learning with controlled probability boundary fence. In ICML. 2024.
[47] Xu, Y., L. Shang. J. Ye, et al. Dash: Semi-supervised learning with dynamic thresholding, In ICMI., pages 11525-11536 2021.
[48] Guo, L., Y. Li. Class-imbalances semi-supervised learning with adaptive thresholding. In ICML, pages 8082-8094. 2022.
[49] Wang, Y., H. Chen, Q. Heng, et al. Freematch: Self-adaptive thresholding for semi-supervised learning. ICLR, 2023.
[50] Li, C., X. Li, Ouyang. Semi-supervised text classification with balanced deep representation distributions. In ACL-IJCNLP, pages 5044-5053. 2021.
12

---

[51] Li, X., Y. Jiang, C. Li, et al. Learning with partial labels from semi-supervised perspective. In AAAL, pages 8666 8674. 2023.
[52] Xu, H., X. Liu, Y. Li, et al. To be robust to be fair: Towards fairness in adversarial training. In ICML, pages 11992-1550.. 2021.
13

---

A Proof of Theoretical Results
Proof. Proof of Theorem 2.1 For any linear classifier f(x) = sign((w,x) x) + b), we first calculate its risk:
(f)
- (1(((x) # #y)
- [1( * y)] + (1- -- (4) ( # v)}+
e_ (1,x [1 (x(x) # +1)] + + [1 (f(x) # =1)]
(2 6- c+ (x.))) [ --
P [/(x) +
[J(x) + -1]
(2 c_ E4) (Fly =
(2 - 6- - e4) .a. - -Pf(x)) +1,y=+1] +1gy
-1] -
= -1)+ +
e_ P_l
-1)-
( (1 a) PP[((x) -1ly -11]= P[f(x)= = ++1
where a Ply = +1]. c PO -ly +1| and c- Pyy -11yy=-1-
Denote x- = [x], ... Mal and w [W], wa] T, we can explicitly calculate R(ʃ++1, and
F[f(x) = +1g = +1] as:
+(f++1)= = (x( -P +b < 0ly +11 P
waz + b < 0]
P[f(x) +1yy + IP
0g +
W.T: + b
where 21, ... ,.d are i.i.d. drawn from Gaussian distributions (N(ss, (at) 1 according to the definition of p in Eq. (2).
Similar to R(f, +1), we have
d
d
R(ʃ, 1)) = P1
1] wii; +b> > 0J, PiJ(x) = 1ly -1y- P P w + b < 0],
where T1.-.. Fd are i.i.d. drawn from Gaussian distributions {N( Ms: {o_) Denote (x) = (w", x) + b² According to the method of (52), we can prove wi
w 1 by con-
tradiction. Based on the properties of Gaussian distribution, R(fest, +1). [[fsa((X) = + + = +1]. R(fast, 1) and (x)
-1y = -1] can be expressed as follows:
Ei Pa)
-b* Eim14 Hi
R(Seat +1) = P Σ
b" b" < 0j = P
(o
Ei ( (
i=d
+ P
Σ
P(]sa((x)= = +1yy +1] P
Fi b" 0] = P
Li=1
mi)
-b* - bi= Mi
E i= w0)
VIE (o وو)
b" + 1
= =1- ①
Σ
Σi= (x: (-w+3)
bb* + Σ d
Σ 14i
R(fosl, 11)
23 6 0 P
E i=d (a (2)
Σ Did ((
6" +
1 Φ
ME
14

---

Li= [zt (-H4))
b" >iz Hi
Pifeat(x)= = - -1yy -1] =
Di + 6" < 0 = IP
Eisl
و (o 1
1=1
b" 1
ME
where & is c.d.f. of normal Gaussian distribution (\ 1). Then, get
+ (0t)00 6-
6" 6ー- ME
(fstl =a(2 c -4+ 1 Σ
+ (1-a)
bO =)=+t)
(b
(b++1) +
(1-a) a1-a
MM
act
We will find the optimal b" which minimizes the overall standard classification error Rest (fsst) by (Ran( = In detail, it is:
taking
db
(Raal dbO
1
1
1
6"
+ 113 -1
=a(2 c- -c)
exp( -
22T
2
Σ
E
1
1 - 12)
1
(1 a2 e- -e4)
expl
v2T
2
ME
ME
(1 -1-o0 a-e 22T
exp( ) 11"
-11 )2)
11
-1
+ af+
1
exp(
1
1 +
13)
1 Σ
0
ME
ME
22
Σ
which can be reformulated as:
6" + 11
10 - 2 2 2log
Ma c- -
ME
(1- -cc - (c4-)))
Denote B log
Ma(2-r-- (2-)) 2c-)) (1-a)(2-2) (2-2e -c--
Without loss of generality, we assume and obtain:
11E2 2(M + 1) + (M2 - 1)ミ M
(12)
Consequently, b" can be given by selecting the smaller absolute value:
(M² +1)+ 1+B (M22 DE?
2n
if > 1,
M
1"
- -(M4+1)a-2MM 1M /2 1+
2p
if 1
M
Then when M > 1, the class-wise standard classification errors are:
R(fsal: +1) = (aaa://)))-aaAA//A///A/A/Ca/a//
R(fsal: -1) = क ( M A (MM
when M < 1, they given by:
R(fsal: +1) = ( + M
R. [[ -1) = (M A / A2 (M,,,, (,;-,(,)),,
where
q(M, a, E4, 2 Ma(2-c -2c
(1-a)[2 2c
A
(M2 - 1)
M
When (0 (o(0)2
Dd (0 = 12 i.e. M 1, Eq. can be rewritten as:
a(2 c_ 22_) (1
4ub² = 2log
2
a2 - 22_ +
15

---

In this case, b* can be expressed as follows:
log al2 a22-e--2ey E
[1-0]12-2e--
2p
and corresponding wass standard classification emrors are given by:
R(ssat; +1)= -21² log( (1-a)(2 2c [4]
al2
22 52
2/15
R(feal: -1) 1b ( -24² + log(
al2-e- -2++)
122
[2-)][2 -c1)
2p
Proof. Proof of Theorem 2.3
According to the results of Theorem 2.1
we can formulate the
class-wise accuracy as:
p ((+)) = 1 - R(Sust, +1), p(-1) = R(Jast, -1).
Accordingly, the variance of class-wise wise accuracy can expressed as:
VCA(fsa)) (ar(p(+1),x(11)) Var(1 R(Jaul, +1); R(Ssat: -1))
Var(R[faal; +1), R(faol: -1)
(R(Jaol +1) R(fect, -1))
2
For convenience, we assume log( a(2-c- (11----204) = 0, and the conclusion will also hold when [1-a)[2-21-
M > max( (I-a)(2-2- -c.]. and M < min( (1 [1- 3c-_.-.11 1). When M 1, it has
a(2-c--26+)
a1226- -2++
a,,,a,c..a Mean Value Theorem, there exists some such that
R[ffss:-1) > R(saat,, +1) because g ar c_, .4) and > Then according Lagrange's
R(Ssol: -1) - R(Sest, +1)
= A / A + + ((M, 4(A - Q
= A / 12 + g(,,, A+ M / q2,
1 exp( (M 1)( (V22 (M, A).
V2T
By analyzing the variation of g(M, a, 64), we can easily verify that R(Jest, -1) R(S +1) increasing when M + co. Similarly, we can prove that R(fest, +1) > R(feel, -1) when M < and R[fset. +1) R(fest, -1) is increasing when M + 0.
16

---

B The training procedure of the model
The Algorithm
! provides a detailed description of the training process of the model.
Algorithm
Training Procedure of 3
Input:
1: Dr: the labeled training dataset
2: Du: the unlabeled training dataset
3: To, To: the number of warm-up epochs, the number of SSMLL training epochs
4: B.: the number of unlabeled batch size
Output: the classifier fw(-).
5: Initialize the classifier parameter W;
6: Warm-up (-( on D, with BAM loss 66 by To epochs;
7: for t = 1 to T₁ do
Ji=1 of Du with Eq. 8: Calculate pscudo-labels [y." N==
9: Estimate {ce] )==K {(1g (o and [(ek (m)
(1) with Eqs. and 10
10: Construct {St, ) k=1
k=K with Eq. 11:
f to [D, BB, do
Optimize fw (-) by minimizing the objective Eq.44 with D;, Du. fy. (ym) and {2p JK=
13: end for
end for
C Time cost comparison
To examine the efficiency of S we perform efficiency comparisons over our S MLL BBAM. SSL baselines (SoftMatch and FlatMatch) and SSMLL baselines (DRML and CAP) on VOC and COCO. Table[]]shows the running time averaged 100 epochs. From Table 5 it can be seen that our method is competitive with the current SSMLL methods the time efficiency and costs less time than the SSL baselines in practice.
Table 5: Time cost (second, s) of each training epoch VOC and COCO.
Method
VOC
COCO
SoftMatch
79.3
726.2
FlatMatch
119.8
1658.1
DRML
4.9
30.4
CAP
28.4
312.5
M'MLL-BBAM
33.1
276.3
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

Justification:
Please refer to 3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully disclose all the information needed reproduce the main experimental results of the paper to the extent.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification:
The code will submitted later.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//uuuii wwwiii/gguie///eeeeeeoeic CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specify all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars were too small to have any visual impact.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
20

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have used a single NVIDIA GeForce RTX 3090 GPU.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No such risks
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "oTlBKT4jvw", "reviewer": "Reviewer_nsG4", "summary": "Based on the traditional binary loss function and negative sampling, when using labeled and pseudo-labeled samples for semi-supervised multi-label learning, there is an issue of variance bias between the feature distributions of positive and negative samples for each label. To solve this problem, authors balance the variance bias between positive and negative samples from the perspective of the feature angle distribution for each label. They also propose an efficient prototype-based negative sampling method to maintain high-quality negative samples for each label.", "questions": "1. In experiments, authors describe the ratio of labeled samples in training as \\alpha, and in Eq. (2), \\alpha is also present, indicating probability. It is recommended that authors provide a more detailed explanation of mathematical symbols to avoid confusion.\n2. In experiments, why were 6 algorithms compared in Table 1, while only 3 comparison algorithms were involved in Table 2?\n3. Is evaluating the algorithm based on three metrics a bit limited in experiments?\n4. In Parameter Evaluation Subsection, the description in 265-266 is confusing: \"And when s is set between 10 and 50, there is no significant change in the performance. One possible reason for this situation is that when the is small, the convergence speed of the model is too slow.\" What does \"the is small\" mean? Is s between 1 and 10 considered smaller?\n5. In terms of methodology, the proposed method trains a multi-label classifier based on binary loss on each label class. Does this overlook the important issue of label correlation in multi-label learning?", "limitations": "The authors mentioned that the limitation of the proposed method lies in the need to improve the experimental results in applications. I consider that whether or not the crucial issue of considering label correlation is effectively addressed is worth pondering.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The definition of the problem scenario is accurate and clear. Their perspective on problem-solving is innovative. According to the experimental results, the proposed method is indeed effective in applications.", "weaknesses": "In terms of writing, the use of some mathematical symbols in the paper is not standardized. In terms of method analysis, the computation complexity of the proposed method is still unclear. In terms of experimentation, the evaluation metrics are not comprehensive enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we would like to thank you for your time and effort in reviewing our submission. Next, we would like to respond to the main concerns raised in the comments.\n\n**Q1: About the some unstandardized mathematical symbols**\n\n**A1**: Thank you for your correction. We change the ratio of labeled samples in training to $\\pi$ and will update it as well as introduce a detailed explanation of mathematical symbols as you suggested in the next version.\n\n**Q2: About the computation complexity**\n\n**A2**: Thank you for your comment, we compare the real running time of our S$^2$ML$^2$-BBAM with CAP and SoftMatch and FlatMatch over VOC2012 and Animals with Attributes2 (AWA) [1], whose numbers of samples are 5717 and 30337, numbers of categories 20 and 85, average numbers of positive classes per instance 1.46 and 30.78. We report the average running time per epoch by running 100 epochs as follows. As shown in the table, the running time of our S$^2$ML$^2$-BBAM is competitive with CAP, and much less than SoftMatch and FlatMatch. We kindly argue that the Gaussian linear transformation, estimating label angle variances and negative sampling are simple, their additional computational cost is very few and they only perform one time per epoch. The next version will discuss the additional computational cost and show efficiency evaluations. \n\n|Method|Average running time per epoch (s) on VOC|Average running time per epoch (s) on AWA|\n|---|---|---|\n|SoftMatch|79.3|268.9|\n|FlatMatch|119.8|542.5|\n|CAP|28.4|109.1|\n|S$^2$ML$^2$-BBAM|33.1|112.3|\n\n**Q3: About the evaluation metrics**\n\n**A3**: Thanks for your advice. We add two new metrics, including **Hamming Loss** and **One Error**. Then, the metrics used to evaluate the performance of multi-label classification cover *example-based* metrics (mAP, Hamming Loss, One Error) and *label-based* metrics (Micro-F1 and Macro-F1) [2]. The following two tables report the experimental results of Hamming Loss and One Error. As shown in two tables, our S$^2$ML$^2$-BBAM also performs better than baselines in most cases. The corresponding results will be updated in the next version.\n\nFor Hamming Loss,\n||VOC||||COCO||||\n|---|---|---|---|---|---|---|---|---|\n|Method|$\\alpha$=5%|$\\alpha$=10%|$\\alpha$=15%|$\\alpha$=20%|$\\alpha$=5%|$\\alpha$=10%|$\\alpha$=15%|$\\alpha$=20%|\n|SoftMatch|0.0594|0.0368 |0.0319|0.0294 |0.0235 |0.0218|0.0211|0.0205|\n|FlatMatch|0.0386 |0.0322|0.0313|0.0290|**0.0227** |0.0213|0.0208 |0.0203|\n|MIME|0.0546|0.0407|0.0336|0.0333|0.0302|0.0265 |0.0250|0.0236 |\n|DRML|0.0564 |0.0518 |0.0377|0.0381 |0.0242 |0.0240 |0.0230 |0.0223|\n|CAP|0.0801|0.0675|0.0622|0.0591|0.0523|0.0512 |0.0499 |0.0558|\n|S$^2$ML$^2$-BBAM|**0.0310**|**0.0259**|**0.0243**|**0.0233**|0.0230|**0.0212**|**0.0206**|**0.0201**|\n\nFor One Error,\n||VOC||||COCO||||\n|---|---|---|---|---|---|---|---|---|\n|Method|$\\alpha$=5%|$\\alpha$=10%|$\\alpha$=15%|$\\alpha$=20%|$\\alpha$=5%|$\\alpha$=10%|$\\alpha$=15%|$\\alpha$=20%|\n|SoftMatch|0.4398|0.1655|0.1308|0.1148|0.1293|0.0948|0.0844|0.0879|\n|FlatMatch|0.1983|0.1366|0.1238|0.1097|0.1215|0.1002|0.0933|0.0878|\n|MIME|0.2099|0.1218|0.0835|0.0949|0.1495|0.1110|0.0883|0.0799|\n|DRML|0.3542|0.2888|0.1720|0.1512|0.1438|0.1288|0.1243|0.1039|\n|CAP|0.1303|0.0918|0.0827|**0.0755**|0.1004|**0.0841**|**0.0788**|**0.0726**|\n|S$^2$ML$^2$-BBAM|**0.1087**|**0.0867**|**0.0817**|0.0795|**0.1000**|0.0878|0.0824|0.0799|\n\n**Q4: Only 3 comparison algorithms were involved in Table 2**\n\n**A4**: Thank you for your comment. We add SoftMatch and FlatMatch as baselines for text datasets, where Back-Translation (very time-consuming) is employed as the strong-augmentation method and none of the weak-augmentation methods is used following [3,4]. The corresponding results are reported in the following two tables. As shown in the two tables, our S$^2$ML$^2$-BBAM consistently outperforms two new baselines across all text benchmarks and metrics, demonstrating its effectiveness. We will update them in the next version.\n\nFor Ohsumed,\n||$\\alpha$=5%|||||$\\alpha$=10%|||||\n|---|---|---|---|---|---|---|---|---|---|---|\n|Method|Micro-F1|Macro-F1|mAP|HammingLoss|OneError |Micro-F1|Macro-F1|mAP|HammingLoss|OneError |\n|SoftMatch|0.4769|0.3056|0.4664|0.0756|0.4213|0.4478|0.2366|0.5106|0.0798|0.5036|\n|FlatMatch|0.5161|0.3073|0.4187|0.0699|0.3943|0.4836|0.2262|0.4751|0.0747|0.4416|\n|S$^2$ML$^2$-BBAM|**0.6671** |**0.6058**|**0.5537**|**0.0467**  |**0.2417**|**0.7100**  |**0.6515**|**0.6345**|**0.0409**  |**0.2186**|\n\nFor AAPD,\n||$\\alpha$=5%|||||$\\alpha$=10%|||||\n|---|---|---|---|---|---|---|---|---|---|---|\n|Method|Micro-F1|Macro-F1|mAP|HammingLoss|OneError|Micro-F1|Macro-F1|mAP|HammingLoss|OneError|\n|SoftMatch|0.3345|0.0612|0.3753|0.0596|0.6630|0.3325|0.0514|0.3949|0.0598|0.6630|\n|FlatMatch|0.3221|0.0519|0.3571|0.0607|0.6629|0.3147|0.0439|0.3706|0.0614|0.6631|\n|S$^2$ML$^2$-BBAM|**0.7057**|**0.5091**|**0.5153**|**0.0262**|**0.1821**|**0.7279**|**0.5825**|**0.5903**|**0.0241**|**0.1500**|\n\n**Q5: Confusing description in 265-266**\n\n**A5**: Thank you for your correction. We want to say that the performance is insensitive to the rescaled norm $s$ when $s\\in [10, 50]$, and the best result is obtained when $s=20$. We will clarify it in the next version.\n\n**Q6: Overlook label correlation**\n\n**A6**: Thank you very much. We agree that label correlation is important to multi-label learning and we should present the overlook of label correlation in Limitations. The label correlation may be helpful to generate high-quality pseudo-labels for unlabeled instances, such as label propagation based on labeled instances and label corrections, as well as regularization term based on label correlation for the classifier weights. We will exploit them in our future work.\n\n**Reference**\n\n[1] Attribute-based classification for zero-shot visual object categorization. IEEE TPAMI, 2013.\n\n[2] A review on multi-label learning algorithms. IEEE TKDE, 2013.\n\n[4] SoftMatch: Addressing the Quantity-Quality Tradeoff in Semi-supervised Learning. ICLR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SxFLYCgtnE", "reviewer": "Reviewer_zC26", "summary": "The paper proposed an interesting idea of using the balanced binary angular margin loss for semi-supervised multi-label learning learning (SSMLL). This is motivated by the empirical observation that the feature distributions of positive and negative samples for each label in SSMLL always suffer from the variance bias problem, and the theoretical results demonstrate that it potentially results in an unfair classifier for SSMLL. To address this issue, the authors propose to balance the variance bias between positive and negative samples, leading to a new and well-designed balanced binary angular margin loss. They also suggest a prototype-based negative sampling technique for efficient training. The idea of this paper is interesting and the proposed method is also well-motivated and well-supported with several experiments on both text and image benchmarks.", "questions": "Please refer to the Weaknesses.", "limitations": "The authors have mostly addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The idea is interesting, and the proposed technique is well motivated and clearly distinguished from prior works.\n2. The variance bias problem is interesting for SSMLL, and the proposed theorems of the intra-class standard classification and the variance of class-wise accuracy can also indicate that the variance bias certainly results in an unfair classifier for SSMLL. This is very solid.\n3. The proposed method is technically sound and it demonstrates a strong performance against current SSMLL methods, especially for Macro-F1 score. In addition, comprehensive experimental results on benchmark datasets clearly demonstrate the effectiveness of the proposed method.", "weaknesses": "1. How does the negative sampling affect the performance of the proposed method? Otherwise, the proportion of positive and negative samples of each category is set as 5. What happens if it is set to bigger or smaller values?\n2. In the experiments, the ResNet-50 model was used as the backbone for image benchmarks. What is the backbone for text benchmarks?\n3. Estimating label angle variances seem to complex and cost most time. Therefore, the efficiency experiments may be performed, such as comparing with the current CAP method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we are very grateful for your time and effort in reviewing this paper. Below are the responses to your questions and comments.\n\n**Q1: How does the negative sampling affect the performance of the proposed method? What happens if the proportion of positive and negative samples of each category is set to bigger or smaller values?**\n\n**A1**: Thank you for your comment. We kindly argue that negative sampling is employed to select high-quality negative samples for each label as mentioned in the introduction and is a widely-used trick in multi-label learning [1,2,3,4]. Besides, we also perform a sensitivity analysis of the proportion ($\\eta$) of positive and negative samples of each category over VOC2012 with $\\alpha$=5% to show the effect of negative sampling on the classification performance. \n\n| $\\eta$ | Micro-F1 | Macro-F1 | mAP    |\n| ------ | -------- | -------- | ------ |\n| 1      | 0.8007   | 0.7439   | 0.7980 |\n| 3      | 0.8007   | 0.7348   | 0.8030 |\n| 5      | 0.8055   | 0.7452   | 0.8004 |\n| 7      | 0.8037   | 0.7411   | 0.7958 |\n| 9      | 0.8024   | 0.7370   | 0.7876 |\n|        |          |          |        |\n\nAs shown in the table, the bigger or smaller values of $\\eta$ will result in a worse classification performance. One possible reason is that the bigger $\\eta$ will introduce many false negative samples and the smaller one will cause the number of negative samples is not enough. We will present the corresponding results and ablation study of negative sampling in the next version, and exploit the selection of $\\eta$ in our future work.\n\n**Q2: What is the backbone for text benchmarks?**\n\n**A2**: Thank you for your correction. We use the bert-base-uncased as the text backbone and will clarify it in the next version.\n\n**Q3: The efficiency experiments may be performed.**\n\n**A3**: Thank you for your suggestion, we compare the real running time of our S$^2$ML$^2$-BBAM with the current semi-supervised multi-label learning method CAP and two semi-supervised learning methods SoftMatch and FlatMatch over VOC2012 and Animals with Attributes2 (AWA) [5], whose numbers of samples are 5717 and 30337, numbers of categories 20 and 85, average numbers of positive classes per instance 1.46 and 30.78. We report the average running time per epoch by running 100 epochs as follows:\n\n| Method           | Average running time per epoch (s) on VOC | Average running time per epoch (s) on AWA |\n| ---------------- | ----------------------------------------- | ----------------------------------------- |\n| SoftMatch        | 79.3                                      | 268.9                                     |\n| FlatMatch        | 119.8                                     | 542.5                                     |\n| CAP              | 28.4                                      | 109.1                                     |\n| S$^2$ML$^2$-BBAM | 33.1                                      | 112.3                                     |\n|                  |                                           |                                           |\n\nAs shown in the table, the running time of our S$^2$ML$^2$-BBAM is competitive with CAP, and much less than SoftMatch and FlatMatch. We kindly argue that the Gaussian linear transformation, estimating label angle variances and negaive sampling are simple, their additional computational cost is very few and they only perform one time per epoch. We will discuss the additional computational cost and show efficiency evaluations in the next version. \n\n**Reference**\n\n[1] Jiang, T., D. Wang, L. Sun, et al. Lightxml: Transformer with dynamic negative sampling for high-performance extreme multi-label text classification. In AAAI, pages 7987–7994. 2021.\n\n[2] Dahiya, K., D. Saini, A. Mittal, et al. Deepxml: A deep extreme multi-label learning framework applied to short text documents. In WSDM, pages 31–39. 2021.\n\n[3] Qaraei, M., R. Babbar. Meta-classifier free negative sampling for extreme multilabel classification. Machine Learning, pages 1–23, 2023.\n\n[4] Liu, Weiwei, et al. The emerging trends of multi-label learning. IEEE TPAMI, 44(11): 7955-7974,  2021.\n\n[5] Lampert, C. H., H. Nickisch, S. Harmeling. Attribute-based classification for zero-shot visual object categorization. IEEE TPAMI, 36(3):453–465, 2013.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MmqJVkkd9Q", "reviewer": "Reviewer_MzwP", "summary": "This paper focuses on the semi-supervised multi-label learning (SSMLL) task and propose a novel and interesting SSMLL method motivated by the variance bias problem, which implies that the variance difference of feature distributions of positive and negative samples for each label in SSMLL is much higher than one in the supervised learning setting. The authors present a theory showing that the variance bias will lead to an unfair classifier, and propose a new balanced binary angular margin (BBAM) loss to balance the variance bias issue from the perspective of feature angle distribution for each label. The performance of the proposed approach is validated on several image and text benchmark datasets, further confirming its superiority.", "questions": "1. Is it feasible to apply the idea of balancing feature angle distributions in other fields, such as imbalanced or long-tailed setting?\n2. Due to the prototype-based negative sampling method's computational demands, does it limit the scalability of the proposed method for other datasets including load of samples or categories?\n3. The proposed BBAM loss relies on the Gaussian assumption for positive and negative feature distribution of each label. This may limit the proposed method’s effectiveness in cases where this assumption does not hold. Can the proposed method extend to other feature distribution cases? Or are there some more general feature distribution assumptions?", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "-\t(Clarity) The paper is well organized and clearly written, the figures are also informative and well-designed.\n-\t(Novelty) The paper is thought-provoking! The variance bias issue within SSMLL and handling it by a new BBAM loss from from the perspective of feature angle distribution are highly innovative. Furthermore, the proposed techniques are also novel as far as I can tell.\n-\t(Quality) The paper is of good quality in my opinion. The algorithm is well designed for balancing the variance bias (via several label-specific linear Gaussian transformations) and efficient negative sampling (via prototype-based negative sample selection). The technical details are all correct as far as I can tell. The empirical evaluation is also comprehensive, covering 4 image and text datasets, and comparing against popular baselines.\n-\t(Significance) The proposed techniques are simple, easy to implement and experimentally highly effective, making the algorithm a strong, potentially impactful baseline for future researchers & practitioners to use and/or improve upon.", "weaknesses": "The linear Gaussian transformations and estimating label angle variances seem to be complex and may cost more time, which could pose challenges for practical applications. Thus, some efficiency experiments are expected to compare the real running time of the proposed method and other baselines, such as CAP. Finally, the reliance on the Gaussian assumption for feature distribution could limit the method's effectiveness in cases where this assumption does not hold.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper. We are encouraged that you agree with the novelty and contributions of our paper. Below are the answers to your questions.\n\n**Q1: Some efficiency experiments are expected to compare the real running time of the proposed method and other baselines. Does it limit the scalability of the proposed method for other datasets including load of samples or categories?**\n\n**A1:** Thank you for your suggestion, we compare the real running time of our S$^2$ML$^2$-BBAM with the current semi-supervised multi-label learning method CAP and two semi-supervised learning methods SoftMatch and FlatMatch over VOC2012 and Animals with Attributes2 (AWA) [1], whose numbers of samples are 5717 and 30337, numbers of categories 20 and 85, average numbers of positive classes per instance 1.46 and 30.78. We report the average running time per epoch by running 100 epochs as follows:\n\n| Method           | Average running time per epoch (s) on VOC2012 | Average running time per epoch (s) on AWA |\n| ---------------- | --------------------------------------------- | ----------------------------------------- |\n| SoftMatch        | 79.3                                          | 268.9                                     |\n| FlatMatch        | 119.8                                         | 542.5                                     |\n| CAP              | 28.4                                          | 109.1                                     |\n| S$^2$ML$^2$-BBAM | 33.1                                          | 112.3                                     |\n|                  |                                               |                                           |\n\nAs shown in the table, the running time of our S$^2$ML$^2$-BBAM is competitive with CAP, and much less than SoftMatch and FlatMatch. We kindly argue that the Gaussian linear transformation, estimating label angle variances and negaive sampling are simple, their additional computational cost is very few and they only perform one time per epoch. Overall, the scalability of our proposed S$^2$ML$^2$-BBAM is well and it can be adapted to other datasets including load of samples or categories. We will discuss the additional computational cost and show efficiency evaluations in the next version. \n\n**Q2: The reliance on the Gaussian assumption for feature distribution could limit the method's effectiveness in cases where this assumption does not hold.**\n\n**A2:** Thank you for your discussion. The Gaussian can be considered as a general tool for many cases. Besides, the linear transformation characteristics of Gaussian lead to simple calculations. We will investigate other distributions such as von Mise-Fisher distribution and GMM in our future works.\n\n**Q3: Is it feasible to apply the idea of balancing feature angle distributions in other fields, such as imbalanced or long-tailed settings?**\n\n**A3**: Thank you for your discussion. We kindly argue that the idea of balancing feature angle distributions can be applied to imbalanced or long-tailed settings, because balancing feature angle distributions can increase the diversity of minor classes and decrease the diversity of major classes to some extent, then address the imbalance issue. We will exploit it in our future works.\n\n**Reference**\n\n[1] Lampert, C. H., H. Nickisch, S. Harmeling. Attribute-based classification for zero-shot visual object categorization. IEEE TPAMI, 36(3):453–465, 2013.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y6saKF1ukq", "reviewer": "Reviewer_ZohP", "summary": "In this paper, the authors investigate the topic of semi-supervised multi-label learning (SSMLL), and find an interesting problem in SSMLL, namely variance-bias issue, which means that the variance difference between positive and negative samples’ feature distributions for each label in SSMLL is much higher than ones in the supervised setting. And two novel theoretical analyses demonstrate that the variance bias will result in the unfairness of the induced multi-label classifier, leading to performance degradation. To address this issue, the authors propose a novel SSMLL method by balancing the variance bias from the feature angle distribution perspective with a new balanced binary angular margin (BBAM) loss. An efficient prototype-based negative sampling method is also suggested to maintain high-quality negative samples for each label. The authors conduct a number of experiments over both image and text benchmarks, and the results show the effectiveness of the proposed method.", "questions": "Please refer to the Weaknesses.", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Interesting motivation: The paper is derived from a novel variance bias problem found by the authors. The authors investigate this problem and point out that it will result in an unfair classifier. A new and easy-to-implement BBAM loss is designed to balance the variance bias for each label and a novel SSMLL method is also proposed based on BBAM loss. The overall motivation is solid, and the proposed method is also reasonable.\n\n- Well-written: The paper is well-written, with clear descriptions of the empirical phenomenon, theoretical analyses, method principles, and experimental design as well as results. The authors introduce the proposed BBAM loss and utilize techniques such as self-training and negative sampling to provide a viable solution, which is presented in a clear and concise manner.\n\n- Experimental results over several image and text benchmarks demonstrate the effectiveness of the method: The paper provides experimental evidence of the proposed algorithm's effectiveness.", "weaknesses": "- Lack of quantitative analysis about the variance difference: The authors perform some extensive experiments on both image and text benchmarks and ablation study to show the effectiveness of the proposed BBAM loss and the corresponding method. It is also expected to perform experiments showing that the proposed BBAM loss can certainly eliminate the variance bias issue.\n- Lack of experiments over benchmarks with more positive classes for each sample: The authors just perform experiments over some benchmarks, i.e., VOC, COCO, Ohsumed and AAPD, where each sample just consists of a few positive classes, and the average number of positive classes is 1~3. Some experiments are expected over benchmarks with more positive classes for each sample, such as AWA.\n- Some minor typos: 1) “\\alpha%” in line 212 should be “\\alpha”. 2) The symbol of indicator function is inconsistent, such as Eq.(9) and Eq.(10). 3) “label angles and both positive …” in line 169 should be “label angles of both positive …”. 4) “Class-Aware Pesudo-” in line 144 should be “Class-Aware Pseudo-”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, we are very grateful for your time and effort in reviewing this submission. We are encouraged that you agree with the contributions of our paper. Below are the responses to your comments.\n\n**Q1: Lack of quantitative analysis about the variance difference.**\n\n**A1:** Yes, adding quantitative analysis about the variance difference will greatly strengthen the paper. Therefore, we compare the variance difference between feature distributions (VDFD) of positive and negative samples of our S$^2$ML$^2$-BBAM during the training procedure with ones of CAP, SoftMatch and FlatMatch, as well as the ablation variant of our S$^2$ML$^2$-BBAM by replacing BBAM with BAM (“w/o BBAM”) on label 6 of VOC2012. The experimental results are summarized as follows: \n\n| Epoch            | 22       | 24       | 26       | 28       | 30       | 32       | 34       | 36       | 38       | 40       |\n| ---------------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| SoftMatch        | 177.43   | 182.69   | 184.28   | 248.53   | 200.90   | 203.44   | 210.61   | 214.32   | 218.90   | 220.28   |\n| FlatMatch        | 119.61   | 156.34   | 173.30   | 148.04   | 166.68   | 171.68   | 169.12   | 169.80   | 173.75   | 179.56   |\n| CAP              | 128.89   | 146.23   | 169.39   | 174.37   | 205.67   | 244.68   | 257.25   | 351.90   | 280.39   | 369.81   |\n| S$^2$ML$^2$-BBAM | **0.42** | **0.74** | **0.83** | **1.09** | **1.28** | **2.94** | **1.76** | **1.72** | **1.98** | **2.08** |\n| w/o BBAM         | 1.13     | 1.33     | 1.51     | 1.84     | 2.36     | 3.25     | 3.32     | 3.62     | 3.45     | 3.62     |\n|                  |          |          |          |          |          |          |          |          |          |          |\n\nFrom the results, the VDFD in the training procedure of our S$^2$ML$^2$-BBAM is much smaller than baselines, which demonstrates that S$^2$ML$^2$-BBAM can effectively balance the variance bias existing in current semi-supervised multi-label learning methods and obtain more fair performance. Otherwise, removing BBAM will increase the VDFD in the training procedure, and this shows the effectiveness of BBAM in balancing the variance bias.\n\n**Q2: Lack of experiments over benchmarks with more positive classes for each sample.**\n\n**A2:** We agree with you and add a new benchmark dataset, **Animals with Attributes2 (AWA)** [1], whose average number of positive classes is 30.78. Here are the experimental results on it:\n\n|                  | $\\alpha$=5% |            |            |              |           | $\\alpha$=10% |            |        |              |           |\n| ---------------- | ----------- | ---------- | ---------- | ------------ | --------- | ------------ | ---------- | ------ | ------------ | --------- |\n| Method           | Micro-F1    | Macro-F1   | mAP        | Hamming Loss | One Error | Micro-F1     | Macro-F1   | mAP    | Hamming Loss | One Error |\n| SoftMatch        | 0.6992      | 0.5476     | 0.6368     | 0.2160       | 0.1580    | 0.6973       | 0.5284     | 0.6524 | 0.2155       | 0.0887    |\n| FlatMatch        | 0.6918      | 0.5221     | 0.6393     | 0.2190       | 0.1029    | 0.6977       | 0.5487     | 0.6459 | 0.2167       | 0.0936    |\n| DRML             | 0.6827      | 0.5399     | 0.6160     | 0.2285       | 0.1360    | 0.6856       | 0.5541     | 0.6246 | 0.2270       | 0.1801    |\n| CAP              | 0.6868      | 0.5742     | 0.6390     | 0.3120       | 0.1146    | 0.7065       | 0.5864     | 0.6415 | 0.2727       | 0.0933    |\n| S$^2$ML$^2$-BBAM | **0.7213**  | **0.5853** | **0.6419** | **0.2091**   | 0.1206    | **0.7255**   | **0.5914** | 0.6463 | **0.2060**   | 0.1103    |\n|                  |             |            |            |              |           |              |            |        |              |           |\n\n|                  | $\\alpha$=15% |            |        |              |           | $\\alpha$=20% |            |        |              |           |\n| ---------------- | ------------ | ---------- | ------ | ------------ | --------- | ------------ | ---------- | ------ | ------------ | --------- |\n| Method           | Micro-F1     | Macro-F1   | mAP    | Hamming Loss | One Error | Micro-F1     | Macro-F1   | mAP    | Hamming Loss | One Error |\n| SoftMatch        | 0.7024       | 0.5524     | 0.6494 | 0.2132       | 0.1494    | 0.7024       | 0.5457     | 0.6518 | 0.2126       | 0.1549    |\n| FlatMatch        | 0.6989       | 0.5507     | 0.6565 | 0.2165       | 0.1116    | 0.7013       | 0.5636     | 0.6577 | 0.2164       | 0.1162    |\n| DRML             | 0.6942       | 0.5727     | 0.6377 | 0.2226       | 0.2609    | 0.6893       | 0.5618     | 0.6338 | 0.2258       | 0.1839    |\n| CAP              | 0.7091       | 0.5905     | 0.6440 | 0.2589       | 0.1045    | 0.7099       | 0.5914     | 0.6451 | 0.2617       | 0.1199    |\n| S$^2$ML$^2$-BBAM | **0.7215**   | **0.5905** | 0.6416 | **0.2109**   | 0.1149    | **0.7279**   | **0.5944** | 0.6476 | **0.2042**   | 0.1188    |\n|                  |              |            |        |              |           |              |            |        |              |           |\n\nIn particular, our S$^2$ML$^2$-BBAM also performs better than baselines in most cases even when there are more positive classes for each sample.\n\n**Q3: Some minor typos.**\n\n**A3:** Thank you for your correction. We will update them and revise the manuscript.\n\n\n\n**Reference**\n\n[1] Lampert, C. H., H. Nickisch, S. Harmeling. Attribute-based classification for zero-shot visual object categorization. IEEE TPAMI, 36(3):453–465, 2013.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
First of all, we sincerely thank all the reviewers for their great efforts in reviewing this submission and providing helpful and valuable comments. Since we cannot revise our paper during the rebuttal period, we plan to make the following revisions in our paper:

- According to most reviewers, we will revise the manuscript and present more experimental details as well as the comparison of the real running time.
- According to Reviewer nsG4, we will introduce new metrics (**Hamming Loss** and **One Error**) in Tables 1 and 2 and include the performance of SoftMatch and FlatMatch in Table 2.
- According to Reviewer zC26, we will include the ablation study of negative sampling and sensitivity analysis of the proportion ($\eta$) of positive and negative samples of each category.
- According to Reviewer ZoHp, we will add the quantitative analysis about the variance difference and a new benchmark dataset, **Animals with Attributes2 (AWA)** [1].

Besides, as suggested by Reviewer nsG4 and Reviewer Mzwp, we will consider introducing label correlation and other feature distribution assumptions and adapting balancing feature angle distributions in other fields as future work.

**Reference**

[1] Lampert, C. H., H. Nickisch, S. Harmeling. Attribute-based classification for zero-shot visual object categorization. IEEE TPAMI, 36(3):453–465, 2013.

Meta-review: All reviewers recommend the acceptance (3 accepts and 1 weak accept), therefore I also believe the paper must be accepted. The authors argue that learning with labeled and pseudo-labeled samples can result in the variance bias problem between the feature distributions of positive and negative samples for each label. To avoid this problem, the paper introduces as method to balance the variance bias between positive and negative samples from the perspective of the feature angle distribution for each label. The authors revised the paper based on the reviewer comments and the reviewers are satisfied with the responses.
Decision: Accept (spotlight)