Multi-Label Open Set Recognition
Authors: Yibo Wang
Jun-Yi Hang
Min-Ling Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: machine learning
multi-label learning
open set recognition
Primary area: other

Abstract: In multi-label learning, each training instance is associated with multiple labels simultaneously. Traditional multi-label learning studies primarily focus on closed set scenario, i.e. the class label set of test data is identical to those used in training phase. Nevertheless, in numerous real-world scenarios, the environment is open and dynamic where unknown labels may emerge gradually during testing. In this paper, the problem of multi-label open set recognition (MLOSR) is investigated, which poses significant challenges in classifying and recognizing instances with unknown labels in multi-label setting. To enable open set multi-label prediction, a novel approach named SLAN is proposed by leveraging sub-labeling information enriched by structural information in the feature space. Accordingly, unknown labels are recognized by differentiating the sub-labeling information from holistic supervision. Experimental results on various datasets validate the effectiveness of the proposed approach in dealing with the MLOSR problem.

Multi-Label Open Set Recognition
Yi-Bo Wang, Jun-Yi Hang, Min-Ling Zhang"
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China {wang-yb, hangiy, zhangml)@seu, edu. cn
Abstract
In multi-label learning, each training instance is associated with multiple labels simultaneously. Traditional multi-label learning studies primarily focus on closed set scenario, i.e. the class label set of test data is identical to those used in training phase. Nevertheless, in numerous real-world scenarios, the environment open and dynamic where unknown labels may emerge gradually during testing. In this paper, the problem of multi-label open set recognition (MLOSR) is investigated, which poses significant challenges in classifying and recognizing instances with unknown labels in multi-label setting. To enable open set multi-label prediction, a novel approach named SLAN is proposed by leveraging sub-labeling information enriched by structural information in the feature space. Accordingly, unknown labels recognized by differentiating the sub-labeling information from holistic supervision. Experimental results on various datasets validate the effectiveness of the proposed approach in dealing with the MLOSR problem.
Introduction
Multi-label Learning (MLL) deals with the problem where an instance can be associated with As practical machine learning paradigm, multi-label learning has been widely applied in various real-world applications, such as image annotation (30
multiple labels simultaneously (37 19
text categorization 266] information retrieval [2]
Traditional multi-label learning studies focus on closed set scenario. That is. they assume that the
class label set of test data is identical to that in the training set [37 8 21 However, in many
real-world scenarios, this assumption rarely holds because the environment is open and dynamic [29]. In addition to the extant label knowledge at training phase, the unknown labels may emerge gradually with the data streams during the testing phase. For example, in Figure the test image is annotated with several relevant labels, some of which are unseen in the training set. The classification task becomes much more challenging because the label correlation between known and unknown labels may degrade the performance of the predictive model. Furthermore, due to the presence of unknown labels in the class label set of test data, these test data are hardly employed in subsequent learning processes, such as incremental learning [15].
Motivated by the potential applications, we formalize a novel framework named multi-label open set recognition (MLOSR), whose goal is to learn multi-label model that can correctly classify known labels for the unseen instance and recognize unknown labels within its relevant label set. This can be regarded as special weakly supervised learning. In MLOSR, the most challenging part is to recognize the unknown labels associated with instances. Since do not have any prior knowledge of unknown labels and they almost always co-occur with some known labels, it is very difficult to separate instances with unknown labels from those with the known labels only.
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

The relevant label set
cloud
building
car
tree
ship
sky
sea trestle bridge
mountain
Figure I: An example. The test image is associated with a variety of relevant labels. Among the set of relevant labels, "cloud", "car". "ship", "building". "tree" and "sky" are known labels seen in the training set, while "sea", "mountain" and "trestle bridge" unknown labels emerging in the testing phase.
Open set recognition (OSR) is paradigm previously proposed in [24] and is formalized risk- minimizing constrained functional optimization problem. OSR describes a scenario where new classes unseen in training occur in testing, thus classifiers must be able to properly identify samples while rejecting unseen ones [9]. (24) proposes a I-vs-Set machine to minimize open set risk by sculpting a decision space from the marginal distances of binary SVM. OSR problem is further studied via algorithm adaptation [ 16). statistical extreme value theory (EVT) 25 32,, margin distribution (33 or hierarchical Dirichlet process [1O]. These existing works based on the fact that each instance owns one ground-trutt label for multi-class cases. Thus, they cannot be used to directly solve the MLOSR problem, due the multiple ground-truth labels in MLL.
To address MLOSR problem, we propose a tailored algorithm named SLAN, i.e. Sub-Labeling informAtion reconstructioN for multi-label open sel recognition. The basic strategy of SLAN is to enrich the sub-labeling information in the sub-label space by leveraging the structural information in the feature space and differentiating it from the labeling information from holistic supervision. Specifically, the underlying structure of feature space is characterized by the sparse reconstruction relationships among training instances. After that, the reconstruction information is utilized to guide the enrichment of sub-labeling information. Then, a unified optimization framework is presented to simultaneously facilitate open set recognizer and multi-label classifier induced with alternating optimization. Our empirical study on datasets from diverse domains demonstrates the effectiveness of the proposed approach.
The rest of the paper organized as follows. We present a brief review related works. Then we formulate the problem and propose the algorithm. Next, experimental results are reported, followed by the conclusion.
2 Related Work
The task of multi-label learning has been extensively studied in recent years [9]. Generally, the major challenge for multi-label learning its huge output space which is exponential to the number of class labels. Therefore, exploiting label correlations has been adopted as a common strategy to facilitate the learning process. Roughly speaking, extant approaches can be grouped into three categories based on the order of correlations, i.e. first-order approaches, second-order approaches and high-order approaches. First-order approaches tackle multi-label learning problem label-by-label manner (2) 33). Second-order approaches exploit pairwise interactions among class labels 7 3). High-order approaches exploit relationships among a subset of or all class labels [4]
OSR is critical for the tasks where incomplete knowledge exists at training time, and unknown classes can submitted to an algorithm during the testing phase, requiring the classifiers to classify the seen classes and deal with unseen ones. According to (9) traditional machine learning methods are adapted to OSR scenario. For instance, SVM-based models add extra constraints on the score space in 24 4 A collective decision-basee model implemented by hierarchical Dirichlet process is proposed in 10 and distance-based models [ are developed by modifying existing classifiers, such nearest class mean classifier and nearest neighbor classifier. Some other approaches focus

---

on the EVT [17. 22 combines the EVT for score calibration with two separated compact abating probability SVMs, where the first SVM is used conditioner and the second SVM fitted yields the posterior estimate, [2] transforms the OSR problem into a set of hypothesis testing problems by modeling the tail part of reconstruction error distribution via EVT. [23] formulates the extreme value machine with distributional information, which can be interpreted by EVT. There are also some
methods trying to incorporate few-shot learning into OSR [5].
MLOSR can be regarded as a combination of MLL and OSR. Thus, straightforward approach to generate a independent recognizer besides the multi-label classifier. However, as unknown labels may co-occur with known labels, it is difficult to separate instances with unknown labels from instances with known labels only, which leads to OSR approaches that could not be applied in MLOSR problems.
Streaming multi-label learning (SMLL)) 31 is similar our MLOSR problem but differs in the setting of unknown labels. It aims to derive a unified model by taking care of the continually emerging new unknown labels on the training data. [3] trains a linear classifier for new labels with the linear hypotheses between labels and classifiers. 28 proposes a novel DNN-based framework to model the
29 presents probabilistic streaming label tree to incorporate new labels, which capture hierarchical correlations among labels. Compared SMLL, MLOSR is much more challenging in recognizing unknown labels as the unknown labels only emerge in testing phase. In the next section, the first attempt towards MLOSR is introduced.
emerging new labels depending on high-order representations.
3 The SLAN Approach
3.1 Problem Formulation
Formally, let X' Rd denote the d-dimensional imput space and y {41, 12.... ................................ .... denote the label space including q class labels. Each multi-label instance can be denoted as (æ; Y.). where 2; E X' is its feature vector and Y, [ y is the set of relevant labels associated with æ,. Here, g-dimensional indicator vector y; = [yk,, M/2, -/iigl is utilized to denote Y,, where yik = indicates class label otherwise. By arranging feature vectors and label vectors of training
& Y and Yiu: instances, we obtain the feature matrix X 21, Fmm] and label matrix Y = [yI,..., ym]]
Given the multi-label training set D {wx, Y,) 1 < S m). the goal of MLOSR is to learn model from D that can correctly classify known labels for the unseen instance and recognize unknown labels within its relevant label set. Conceptually, given the multi-label training set D, an open space risk function Ro and an empirical risk function Rg, multi-label open set recognition aims to derive a measurable recognition function f E H by minimizing the following Open Set Risk:
argmin Ro(f) + (x JEH
where X, is a regularization constant.
3.2 Structural Information Discovery
To characterize the underlying manifold structure of feature space, weighted directed graph G (V, E, S) constructed over the set of training instances, where = (a, 0 S i I m} corresponds to the set of vertices and È = {xx, 2;, St) + 0, < i * j I m] corresponds to the set of edges from 2; to æ, with nonzero weight.
Furthermore, S = [ mx corresponds to the weight matrix encoding the relationships among all training instances. Conceptually, the weight value si reflects relative importance of x; in reconstructing æy. Thus, by implementing global sparse reconstruction, the weight matrix S is instantiated by solving the following optimization problem:
min S
XS
X| + Po||S||
s.t.
$; = 0, V1 < i m
Here, the first term controls the linear reconstruction error via squared Frobenius norm while the second term controls the sparsity of reconstruction via & norm. The relative importance is balanced
3

---

by the trade-off parameter No- Then, the constrained optimization problem in Eq. 2 can be solved via a standard ADMM (Altemating Direction Method of Multiplier) procedure
3.3 Sub-Labeling Information Enrichment
According the manifold assumption, the structural relationship specified in the feature space should also be preserved in the entire label space to enrich labeling information originally encoding in the indicator vector y:
13 38.. That is, y: can transformed into a numerical labeling vector
z, [2.1, 2.2,...: Zigl under holistic supervision which encodes richer semantics for predictive
model induction.
However, this assumption might be suboptimal in the entire label space. Considering one specific label & E V. let f = [/ ................................ fi J. 1 [tg] represent the enriched sub-labeling information of xz. With the manifold assumption, the structural information would be maintained in the sub-label space )" [4+). Then, instances with le can be reconstructed via the instances without l as well as the enriched sub-labeling information. Nevertheless, these reconstructed instances can not be assigned with 4 since there's not enough positive labeling information w.r.t. 4g- That is, the sub-labeling information is differentiated from labeling information with holistic supervision. Thus, in open set scenario, if ( is specified as an unknown class label, such difference can be employed as a criterion for one specific instance distinguish whether it associated with an unknown class label.
Let concatenate all fhh denoted by F;-- = - E R(4-1)xm and all zi, denoted by Z (21,-.., Zm] E R.xmm The enriched sub-labeling information is generated via leveraging the structural information encoded in S by solving the following optimization problem:
9
min Z, F Fi....FF
2
Us
Σ
(3)
9
1 11
Here, 8f is indicator variable, where 6,- if L. is not associated with æ;; otherwise 8 = 0. P is (g 1) x q projection matrix align 2, with f without regard to lg which removes the k-th row of the identity matrix. The first term supervises the labeling information from the holistic aspect. The second term conveys the manifold structure specified in the feature space to the sub-label space. The third term minimizes the labeling information difference between sub-label space and entire label space to differentiate instances with or without such unknown class label, which implicitly reduces the open space risk.
During the testing phase, for an unseen instance D., the reconstruction coefficients w.r.t. la is identified by resorting to the ADMM technique over the training set. After that, the enriched sub- labeling information f of a. is determined via the second term of 3). Thereafter, whether unknown labels are associated with x. is determined by the following recognizer G via ensemble majority voting:
Junknown, if Ei=100 9k I 0,
G(x..) =
known, if Ei= Ik (X > O.
where gi (xx.) -1 if P > Ps: otherwise gx(x)) = 1. pk is the threshold, and can be
chosen so that 100x instances with 4 in training set satisfy |1 - P PP > Pk. The labeling information 2, under holistic supervision is generated by the following MLL classifier.
3.4 MLL Classifier Training
Similar to the existing strategy in previous OSR algorithms, instead of training the classifier indepen- dently, we perform open set recognizer and multi-label classifier induced simultaneously. Then, the sub-labeling information can be optimized by considering both the manifold assumption and model outputs. Furthermore, considering the label correlation between known and unknown labels, such jointly optimization procedure with recognizer can facilitate the classifier more robust. We denote W 199d and b E [ multi-label classifier and adopt the squared Frobenius norm as the

---

regularization term to control the model complexity:
M
Wa, + b 2:l12 +
P1 2
|
(5)
Let B& = -b (g 1) denote the indicator matrix with byf 8.
Then, the objective function of
the unified framework is shown as follows:
min W.b.Z.
2
|I
Fulli + I o (PaZ Fx)l
F Fi.....F F
(6)
D1 7 Z YIF 1 (WX
2
M1
W
2
Here, The first two terms control the open space risk and remaining terms control the empirical risk.
3.5 Alternative Optimization
Update Z With fixed F1, Fg. W and b, the optimization problem Eq. can be stated as follows:
min Z
z - (WX + b1, I Y
2
a
B&
- -
2
A=1
The above optimization problem can be solved by updating Z with gradient descent. The gradient of the objective function w.r.t.
VZ = IZ - (WX + +b,,)) + (Z
(8)
+ a _ P (P - Fx))
Update FI...., Fg With Z. W and b fixed, the optimization problem Eq. can be stated as follows:
8 F&S
min
F BB (P FF)
(9)
Ft 2
Similarly, gradient descent is employed, and the gradient w.r.t. F3 is:
VF, FF,T + aB, (F, P₂Z)
(10)
where T (S Lm I
Update W and b While and Fi...., F, are fixed, the optimization problem (4) can be stated as follows:
min tr (r(EETE)) + Mntr +utrr(WW (WWT) W.b
(1I)
S.t. Z- WX
b1 m + E
Here, E = [ei,..., Cm) Ryxm where e; of the predictive model, a kernel extension is further facilitated for the general nonlinear case. Let & (o(x1)..... p(=m)] E bb where o( Rd + R² corresponds to the feature mapping that maps the feature space to some higher dimensional Hilbert space with à dimensions. Then, the Lagrangian function of this problem can be formulated as:
(Wa; + b). To achieve better performance
L(W, b, E, A) tr(EET) + pitr(WW))
(12)
- tr (W
Ar(AA(W&:1)) + b1 m + Z))
5

---

Algorithm The pseudo-code of SLAN
Input:
The multi-label training set D, the trade-off parameters a; B, Y, M1, 7, unseen instance æ;,;
Output:
The predicted label set Y. for w., the recognition result G(r..).
Process:
Instantiate the weighted graph G (V, E,S) by solving Eq. ( with ADMM procedure:
2: Calculate the kernel matrix K = [,( 2;j mmx
3: Initialize Z with Y:
4: Initialize F& with P&Y (1 < 5 !)
5: repeat
6: Update Z according to Eq. 7):
7: Update F2 according to Eq.
8: Update W and b according to Eq. 11)
9: until convergence or maximum number of iterations being reached
10:
return Y, and ((x.) according to Eq. (1) and Eq. 4
where = [axd]] E IR9x stores the Lagrange multipliers. According to the KKT conditions, we obtain:
b ZH $1m
1 m
(13)
(Z b1 T H
where H LK Imxm and K
with its element Ri x((x,, xby) = ((x,) ((x,)
based on the chosen kernel function x(.,)) -). Then, by incorporating the specified kernel function, the modeling output is denoted by 1 AK + b1 Furthermore, given an unseen instance x, E X' its
n1
relevant label set is predicted as:
Y. U1x ouik(E,, 0,0 4
(14)
1=1
The pseudo-code of SLAN is summarized in Algorithm Given the multi-label training set, weighted graph is constructed to characterize the manifold structure of feature space (Step 1). After that, the alternative optimization strategy is adopted to optimize open set recognizer and multi-label classifier simultaneously (Step 2-9). Finally, the relevant label set of the unseen instance is predicted and the recognition result is generated based on the learned model (Step 10).
Table Characteristies of experimental data sets.
Dataset
[S]
dim((S
L(S)
LCurd(S)
LDen(S)
DL(S)
PDL(S)
llog
1208
925
17
0.966
0.057
96
0.079
enron
1702
1001
24
3.124
0.130
548
0.322
slashdot
3659
1079
14
1.173
0.084
119
0.033
recreation
606
15
1.361
0.091
259
0.052
core15k
5000
499
44
2.214
0.050
1037
0.207
arts
5000
462
14
1.512
0.108
314
0.063
education
5000
550
11
1.374
0.125
173
0.035
revsubset2-2
6000
944
39
2.170
0.056
489
0.082
bibtex
7395
1835
0.954
0.035
380
0.051
4 Experiments
4.1 Experimental Setup
Table summmaiizes the detailed characteristics each benchmark multi-label data set S employed in the experiments, including the number of instances (S), number of features dim(S), number of class labels T.(S,, label cardinality I.Card(S). label density I.Den(S,, number of distinct label sets DL(S) and proportion of distinct label sets PDL(S). To alleviate the influence of extreme

---

Table 2: Experimental results of each compared approach (mean--std) with different label batch size (denoted by #label). The best and the second best performance of each data methods highlighted in boldface and underline respectively. In addition, // indicates whether SLAN is statistically superior/inferior the comparing approaches on each data set with pairwise t-test (at 0.05 significance level).
Dataset
#label
LIFT
MUENLPLR
SENCE
LIMIC
SLAN
Ranking loss (the smaller, the better)
0.339-+0.033
0.400=:0.0255
0.335--0.039
0.350=+00035
0.344+0.0000
0.358+0..00
0.418+00.026
0.357-+0001
0.367+0.040
0.359++0.033
llog
0.365110.038
0.42110.025
0.362:10.037
0.373-100337
0.366100.033
0.368=t0.0022
0.425=00.022
0.366=100333
0.376=000333
0.370=0.003
0..068:-0.028
0.426+t0.019
0.367=:0029
0.377=+0.332
0.374++0.030
0.174::0.02
0.236+0.024
0.159
0.199+00031
0.157+0.016
0.179::0.014
0.241+00011
0.172+000099
0.194+0.011
0.169+0.011
enron
0.179110.01
0.245L1..013
0.104
0.19310.012
0.172.10.012
12
0.180+0.011
0.245+00.012
0.174
0.193=0.012
0.174=0.010
0.237100013
0.332
0.214
0.27910.018
0.214
0.25810.0122
0.343+0.015
0.222
0.28610.017
0.247
recreation
0.260+0.0113
0.346+0.015
0.247=0.019
0,287=0.018
0.052
0.264
0.34710.015
0.018
0.286110011
1.258L0.001
0.312+:0.026
0.404
0.226
0.324+0.028
1.29--0.022
0.324
0.412+0.019
0.009
0.337
0.260-0.022
slashdot
0.335100.019
0.41810.011+
0.30010.015
0.350-0.025$
0..081
0.335=00021
0.418+0.011
00.01
0.001
0.270
0.006
0.317
0.0062
0.267=00045
0.240
0.231+00.020
0.332+00018
0.221-000222
0.287+0.051
0.266+0.013
corel5k
12
0.239--0.017
0.337+0.019.
0.231-
0.292+0.055
0.276+00011
17
0.20710.018.
0.342.0..01
0.238
0.29510.005
0.28310.011
22
0.252+t0.018
0.12
0.243+0.020
0.299=0.052
O. 0.001
Ü
0.220
0.3050000300
0.193
0.297
0.187
0.324+00.04
0.376+0.040
D.311+0.05
0.397+0.076
0.296+00.04
0.348+0.029
0.392+0.029
0.335+0.033
0.421
0.318+0.030
1..036.
0.40920.0027
0.351
0.401
0.332-100034
0.222+:0.028
0.303+00.34
0.002
0.245
0.085-00009
education
0.300:10.059
0.354
0.208
0.34410.076
0.270-1.0049
0.311+00033
0.365+0.037
0.290-0.0522
0.356+0.046
0.281
0.1201100055
0.25910.012
0.09310.005
0.226
0.097
0.220=10.035
0.323+0.027
0.197=00033
0.279=0.042
0.197=0.025
revsubset2-2
0..23--:0.030
0.333+0.024
0.212
0.211+0.027 0.0022
0.290+0.049
0.248+10.020
0.33910.016
0.22610.020
0.224
0.263
0.350-0.0117
10.021
0.004
0.208
0.217110.026
0.396100000
0.20000.0500
0.26710.013
0.114
0.2141:0033
0.387:00013
0.2050
0.262=0.024
0.114=000017
bibtex
0.216-H0.035
0.388+
0.247+000533
0.264
0.117+000017
11
0.2211
0.39110.016
0.250_10.053
0.270.1.0026
0.122.10.020
14
0.223
0.39010.0000
0.202400.044
0.272.10.019
0.123110.016
imbalance, any class label with rare appearance or with overly-high imbalance ratio is excluded from the label space following previous settings 344.
For given dataset, we randomly select 50% labels as known labels and the remaining labels as unknown labels with different label batch sizes. Then, sample 60% instances without unknown labels to form training set while the remaining instances are treated as test data. The sampling procedure repeated ten times, and the mean metric value as well standard deviation for each label batch are reported.
To evaluate the performance of multi-label classifiers, utilize five widely-used multi-label eval- uation metrics (37), including Ranking loss, One-error, Coverage, Average precision and Macro- averuging AUC. In addition, F-measure employed to evaluate the performance of recognizers.
To validate the effectiveness of the proposed SLAN approach multi-label learning, four multi-label learning approaches are used for comparative studies.
A multi-label learning approach, which induces classifiers with the label-specific features generated via conducting clustering analysis for each class label. [parameter configuration: 0.1]
LIFT 35
7

---

Table 3: Experimental results of each compared approach (mean--std) with different label batch size (denoted by #label). The best and the second best performance of each data methods highlighted in boldface and underline respectively. In addition, // indicates whether SLAN is statistically superior/inferior the comparing approaches on each data set with pairwise t-test (at 0.05 significance level).
Datasel
#label
OC-SVM
IFOREST
MUENLFOREST
SLAN
F-measure (the greater, the better)
3
000013
0.494
0.340=0.067+
0.672-0.058
5
0.440-0.0644
0.436=10.000
0.309=0.067
0.576:10.062
llog
0.391=00.449 7
0.387=00046
0.281=0.057
0.501+0.005
9
0.362=00.047
0.359=t0.043
0.267=00.05
0.456+00.043
6
0.352=#0 100
0.350=00099
0.398=0.116
0,406
enron
9
0.283=00.00
0.282= 108
0.314=0.116
0.329
0.251+0.113e 12
0.250+0.1122
0,276+0.120
0.281+0.099
4
0.34610.0066
0.33910.0066
0.31810.054
0.376.10.066
recreation
0.283+0.0022 6
0.279=0.052
0.265=0.044
0.305=00.055
8
0.230-0.033
0.227=0.035
0.218+0.033
0.244+0.003
3
0,411 100
0.409
0.31020.088
0.52810.127
slashdot
5
0.350=00.044
0.349
0.274=0.0088
0.428
0.293+0.007 7
0.293+0.072
0.235+0.005
0.350-+0.069
7
0.496
0.48910.016
0.32710.013
0.65310.015
12
0.430=0.002-
0.424
0.298=00.013
0.509
corel5k
17
00.08-
0.373+00018
0.272+00.00
0.462++0.012
22
00.048
0.340+0.017
0.253+0.009
0.414+00015
3
0.356-0.004
0.354
0.34010.058
0.39510..105
arts
5
0.203+0.040
0.207
0.264=000033
0.288
7
0.200-00.044
0.229-10.02
0,224-10.038
0.235-10.047
4
0,28240:086
00,02
0.27420,082
0.277=0.0059
education
6
0.230+00.000
0.231 0.000
0,226++0.057
0.224+00040
8
L0.09
00.031
0.38710.017
0.616-0.0025
12
0.408
0.414
0.346=0..02
0.518
revsubse12-2
16
0.304
0.370
0.315=0.014
0.452
20
0.341+00.02
0.308+
0.291+00015
0.404+00011
5
0.367100002
0.0000000000
0.478
0.684
8
0.526+00.028
0.024
0.450=0.027
0.621+0.024
bibtex
11
0..09+0.0226
0,487=0.024
0.424=0.024
0.563+00.02
0..00- 14
0.459
0.403-L0.022
0.518-L0.029
MUENLPLR SVM-based dynamic multi-label learning approach which trains a of linear classifiers by minimizing misclassification loss and pairwise ranking loss. [parameter
configuration: C1 = C2 =
SENCE (7]: multi-label learning approach based on label-specific features generated by cixture-based clustering ensemble. [parameter configuration: 0.4].
LIMIC 21: A multi-semantics multi-label metric learning approach coupled with ML-
LNN 36 which learns one global and multiple label-specific local metrics simultaneously.
[parameter configuration: \₁ 1, A2 = 100, 7 2, K 10].
With the first attempt towards solving the MLOSR problem, there is method can be directly applied. Thus, we compare the proposed SLAN approach with existing anomaly detection approaches.
SVM-based approach which constructs a hyper-sphere surrounding all instances from known labels.
OC-SVM 20
An unsupervised forest-based anomaly detection approach which employs average path length over all trees as the anomaly score.
IFOREST 18
MUENLFOREST 39]: A forest-based dynamic multi-label learning approach which utilizes
clustering process in each nodes by considering the feature space and the label patterns. [parameter configuration: 5, &b 256, 100, em = 9].
For the proposed SLAN approach, trade-off parameters are set as a = 0.1, 0.1, 10, pr 0.1, = 0.8. No is fixed to be 0.1. The sensitivity analysis of parameter configurations is conducted in Subsection 4x
Linux server equipped with Intel Xeon CPU (48 cores @ 2.67GHz) and 256GB memory is used for supporting the experiments.

---

4.2 Experimental Results
The detailed experimental results in terms of Ranking loss and F-measure are reported in Table 23 Due to the page limit, the results on other metrics are shown in the Appendix. Meanwhile, pairwise t-test [6] is conducted to demonstrate whether the performance of SLAN statistically superior/inferior to the comparing approaches each data set. The resulting win/tie/loss counts are summarized in the supplementary material.
Based on the reported experimental results, the following observations can be made:
Compared with the performance on close set instances (#label = 0), the performance of all comparing approaches on open set instances degrades. Nonetheless, across all multi-label evaluation metrics, SLAN achieves superior or least comparable performance against the comparing approaches in 91.7% cases. The results clearly indicate the jointly optimization with recognizer serves a more effective way to achieve more robust multi-labe classifier in the open environment.
Comparing with anomaly detection approaches, SLAN achieves better performance in 83.3% cases. Possible reasons that: (a) OC-SVM and IFOREST previously designed for multi- class scenario which can not directly solve MLOSR problem. (b) For the construction of MUENLFOREST, instance augmented with its predictive values derived from MUENLPLA, However, the predictive values might be suboptimal as MUENLPLR is trained without considering open space risk.
In the multi-label setting, instances with unknown labels may share the same dense region of instances with known labels, which makes multi-class anomaly detection approaches tend reject instances with unknown labels. That is why since MUENIPLR is under multi-label setting, it still inferior to FFREST and OC-SVM.
4.3 Parameter Sensitivity Analysis
In this section, study the sensitivity analysis of trade-off parameters a, B, 7, M1, shown in Algorithm[] | Figure [ illlustrates how the performance of SLAN changes with varying parameter configurations on data set enron. As shown in Figure SLAN achieves relatively stable performance on multi-label metrics and somewhat sensitive on F-measure. this paper, trade-off parameters are set as 0.1, 3 0.1,7 7 10, N1 0.1, 0.8, which can be employed the default parameter setting.
(a) Varying &
(b) Varying 3
(c) Varying
(d) Varying 14n
(c) Varying T
Figure Performance SL.NN with varying value of trade-off parameters on enron.
Conclusion
The major contributions of our work two-fold: 1) We formalize novel learning framework named multi-label open set recognition (MLOSR), which aims to classify and recognize instances with unknown labels in multi-label setting, suggesting new direction for multi-label learning. 2) We propose a novel MLOSR approach named SLAN which can facilitate open set multi-label classification by utilizing sub-labeling information and recognize the unknown labels by differentiating the sub- labeling information from holistic supervision. Extensive experimental results clearly validate the effectiveness of the proposed SLAN approach.
However, SL.AN enriches #labels+1 sub-labeling information, which could hardly generalize to extreme multi-label data set. Meanwhile, SLAN works in the multi-label learning schema where

---

the feature representations of instances may less informative. In the future, it is interesting to investigate towards extreme multi-label learning achieve tolerable scalability and design deep ML.SSR approaches with discriminative feature representations. Furthermore, it is desirable to extend evaluation metries for MLOSS.
Acknowledgments
The authors wish thank the anonymous reviewers for their helpful comments and suggestions. This work was supported by the National Science Foundation of China (62225602) and the Big Data Computing Center of Southeast University.
References
[1] Abhijit Bendale and Terrance Boult. Towards open world recognition. Proceeding of 2015 IEEE Conference Computer Vision and Pattern Recognition, pages 1893-1902, Boston, M.A. 2015.
[2] Matthew R. Boutell, Jie bo Luo, Xi-Peng Shen, and Christopher M. Brown. Learning multi-label scene classification. Pattern Recognition, 3709)1157-1771, 2004.
[3] Christian Brinker, Eneldo Loza Mencía, and Johannes Fürnkranz. Graded multilabel classifica- tion by pairwise comparisons. In Proceedings of the 14th IEEE International Conference on Data Mining, pages 731-736, Shenzhen, China, 2014.
[4] Hakan Cevikalp. Best fitting hyperplanes for classification. IEEE Trans. Pattern Anal. Mach. Intell., 1966)1076 2017.
[5] Yongjuan Che, Yuexuan An, and Hui Xue. Boosting few-shot open-set recognition with multi- relation margin loss. In Proceedings of the 32th International Joint Conference on Artificial Intelligence, pages 3505-3513, Macao, SAR, China, 2023. ijcai.org.
[6] Janez Demsar. Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res., 7:1-30, 2006.
[7] Johannes Fürnkranz, Eyke Hüllermeier, Eneldo Loza Meneía, and Klaus Brinker, Multilabel classification via calibrated label ranking. Machine Learming, 73(2):133-153, 2008.
[8] Yi Gao, Miao Xu, and Min-Ling Zhang. Unbiased risk estimator to multi-labeled complemen- tary label learning. In Proceedings of the 32th International Joint Conference on Artificial Intelligence, pages 3732-3740, Macao, SAR, China, 2023. ijeai.org.
[9] Chuan-Xing Geng, Sheng-Jun Huang, and Song-Cas Chen. Recent advances in open set recognition: A survey. IEEE Trans. Pattern Anal. Mach. Intell., 43(10):3614-3331 2021.
[10] Chuanxing Geng and Songean Chen. Collective decision for open set recognition. IEEE Trans. Knowl. Data Eng., 39(1):1112222000 2022.
[11] Euhanna Ghadimi, André Teixeira, Iman Shames, and Mikael Johansson. Optimal parameter selection for the alternating direction method of multipliers (ADMM): quadratic problems. IEEE Trans. Autom. Control., 6043),644-658, 2015.
[12) Siddharth Gopal and Yiming Yang. Multilabel classification with meta-level features. In Proceeding of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 315-322, Geneva, Switzerland, 2010.
[13] Peng Hou, Xin Geng, and Min-Ling Zhang. Multi-label manifold learning. In Proceedings of the 30th AAAI Conference on Artificial Intelligence, pages 1680-1686, Phoenix, AZ, 2016.
[14] Ming Huang, Fu-Zhen Zhuang, Xiao Zhang, Xiang Ao, Zheng- Yu Niu, Min-Ling Zhang, and Qing He. Supervised representation learning for multi-label classification. Machine Learning, 108(5):747-763, 2019.
10

---

[15] Sheng-Jun Huang and Zhi-Hua Zhou. Active query driven by uncertainty and diversity for incremental multi-label learning. Hui Xiong, George Karypis, Bhavani Thuraisingham. Diane J. Cook, and Xindong Wu, editors, Proceedings of the 13th IEEE International Conference on Data Mining, pages 1079-1084, Dallas, TX, 2013.
[16] Pedro Ribeiro Mendes Júnior, Roberto Medeiros de Souza, Rafael de Oliveira Werneck, Bernardo V. Stein, Daniel V. Pazinato, Waldir R. de Almeida, Otávio A. B. Penatti, Ricardo da Silva Torres, and Anderson Rocha. Nearest neighbors distance ratio open-set classifier. Mach. 2017.
Leamn, 106(3):359-333)
[17] Samuel Kotz and Saralees Nadarajah. Extreme value distributions: theory and applications. World Wcientific, 2000.
[18] Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In Proceedings of the 8th IEEE International Conference on Data Mining, pages 413-422, Pisa, Italy, 2008
[19] Wei-Wei Liu, Hao-Bo Wang, Xiao-Bo Shen, and Ivor W. Tsang. The emerging trends of multi- label learning. IEEE transactions on pattern analysis and machine intelligence, 44(11):7955- 7974, 2021.
[20] Jun-Shui Ma and Simon Perkins. Time-series novelty detection using one-class support vector machines. In Proceedings of the 2003 International Joint Conference on Neural Networks, volume 3, pages 1741-1745, Portland, OR, 2003.
[21] Jun-Xiang Mao, Wei Wang, and Min-Ling Zhang. Label specific multi-semantics metric learning for multi-label classification: Global consideration helps. In Proceedings of the 32th International Joint Conference on Artificial Intelligence, pages 4055-4063, Macao, SAR, China, 2023.
[22] Jesse Read, Bernhard Pfahringer, Geoff Holmes, and Eibe Frank. Classifier chains for multi- label classification. Machine Learning, 85(3):333-355 2011.
[23] Ethan M. Rudd, Lalit P. Jain, Walter J. Scheirer, and Terrance E. Boult. The extreme value machine. IEEE Trans. Pattern Anal. Mach. Intell., 40(3):762-768, 2018.
[24] Walter J. Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E. Boult. Toward open set recognition. IEEE Trans. Pattern Anal. Mach. Intell., 35(7):1577-1772, 2013.
[25] Walter J. Scheirer, Lalit P. Jain, and Terrance E. Boult. Probability models for open set recognition. IEEE Trans. Pattern Anal. Mach. Intell., 36(1 1):2317-2324, 2014.
[26] Pingjie Tang, Meng Jiang, Bryan (Ning) Xia, Jed W. Pitera, Jeffrey Welser, and Nitesh Chawla. Multi-label patent categorization with non-local attention-based graph convolutional network. Proceedings of the 34th AAAI conference on artificial intelligence, pages 9031, New York, NY, 2020.
[27] Yi-Bo Wang, Jun- Yi Hang, and Min-Ling Zhang. Stable label-specific features generation for multi-label learning via mixture- based clustering ensemble. IEEE CAA J. Autom. Sinica, 9(7):1248 121-1124-1,,,, 1261, 2022.
[28] Zhen Wang, Liu Liu, and DaCheng Tao. Deep streaming label learning. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pages 9963-9972, Virtual Event, 2020.
[29] Tong Wei, Jiang-Xin Shi, and Yu-Feng Li. Probabilistic label tree for streaming multi-label learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1801-1811, Virtual Event, Singapore, 2021.
[30] Ren-Chun You, Zhi- Yao Guo, Lei Cui, Xiang Long, Ying-Ze Bao, and Shi-Lei Wen. Cross- modality attention with semantic graph embedding for multi-label classification. In The 34th AAAI Conference on Artificial Intelligence pages 17709-12716. New York, NY, 2020.
[31] Shan You, Chang Xu, Yun-He Wang, Chao Xu, and Da-Cheng Tao. Streaming label learning for modeling labels on the fly. CoRR, abs/1604.05449, 2016.

---

[32] He Zhang and Vishal M. Patel. Sparse representation based open set recognition. JEEE Trans. Pattern Anal. Mach. Intell., 39(8):1600-1696, 2017.
[33] Min-Ling Zhang, Yu-Kun Li, Xu-Ying Liu, and Xin Geng. Binary relevance for multi-label learning: An overview. Frontiers of Computer Science, 12(2):91-2022, 2018.
[34] Min-Ling Zhang, Yu-Kun Li, Hao Yang. and Xu- Ying Liu. Towards class-imbalance aware multi-label learning. IEEE Trans. Cybern,, 52(6))4459-444 4471, 2022.
[35] Min-Ling Zhang and Lei Wu. Lift: Multi-label learning with label-specific features. FEEE Trans. Pattern Anal. Mach. Intell., 37(1):107-120, 2015.
[36] Min-Ling Zhang and Zhi-Hua Zhou. ML-KNN: A lazy learning approach multi-label learning. Pattern Recognit., 40(7):2038-208) 2007.
[37] Min-Ling Zhang and Zhi-Hua Zhou. A review on multi-label learning algorithms. IEEE transactions on knowledge and data engineering, 1618):181-11837, 2013.
[38] Qian- Zhang. Yun Zhong, and Min-Ling Zhang. Feature-induced labeling information enrichment for multi-label learning. In Proceedings of the 32th AAAI Conference Artificial Intelligence, pages 4446-4453 4453, New Orleans, LA, 2018.
[39] Yue Zhu, Kai Ming Ting, and Zhi-Hua Zhou. Multi-label learning with emerging new labels. IEEE Trans. Knowl, Data Eng., 30(10):1901-1114 2018.
12

---

A Appendix
A.1 Experimental Results
Table 5 7 and 9 report detailed experimental results in terms of Average precision, Macro- averaging A C, Coverage and One-error, which covered in the Experimental Results part of the main body due to page limit. Besides, the pairwise t-test is conducted to demonstrate whether the performance of SLAN is statistically superior/inferior the comparing approaches on each data set. The resulting win/tie/loss counts in terms of multi-label evaluation metrics are summarized in Table 6 as well F-measure Table
Table 4: Experimental results of each compared approach (meantsstd) with different label batch size (denoted by #label). The best and the second best performance of each data set methods are highlighted in boldface and underline respectively. In addition, /o indicates whether SLAN is statistically superior/inferior the comparing approaches each data set with pairwise t-test (at 0.05 significance level).
Dataset
#label
LIFT
MUENLPLR
SENCE
LIMIC
SLAN
Averuge precision (the greater, the better)
0.496:10.035
0.440=0.023
0.506=0.044
0.478=0.035
0.497=0.033
0.479-10.035
0.425++00226
0.487=:0.044
0.466++0.033
0.484+0.033
llog
0.473-+00005
0.421+0.025
0.480==0.041
0.460+00030
0.477-00033
0.469-10.035
0.416.10.022
0.048
0.45910.027
0.473
0.469=000029
0.413+0.021
0.476=0.034
0.459=0.024
0.469
0.727
0.663+:.054
0.747110.037
0.695
0.751
enron
0.67410.031
0.612:20.033+
0.682:10.032
0.653#0.031
0.687
0.659110.335
0.59510.0333.
0.66710.033
0.641:0.026
0.669-0.033
12
0.655=0.0255
0.591
0.663+0.023
0.638#0.020+
0.664=0.025
0.685110.017
0.57410.019
0.718
0.71710.026 0.619
0.632.10.033
recreation
0.66120.012
0.562+00.177
0.684
0.681=0.023
0.65911.015
0.559:10.11
0.67811.0222
0.61810.026
0.673:0.000
0.653+00.01
0.556+0.016
0.622
0.615=0.025
0.666-00.018
0.563100024
0.46210.022
0.5901
0.641 0.342
0.553+0.018
0.456:0.018
0.587=0.020
0.534#0.029
0.628=10.09
slashdot
0.541:0.016
0.450+0.0133
0.575=0.015
0.520=0.025
0.617=0.026
0.541+0.018
0.449+0.0012
0.074
0.518+0.020
0.614+-0.023
0.46310.0111
0.33810:012
0.010:
0.383.0.0533
0.471
0.426=100019
0.320+0.014+
0.421-0.0220
0.370=0.057
0.431
core15k
0.417#0.014
0.316+0.016
0.119-
0.367=0.055
0.417
17
0.411-0.011
0.313+0.011
0.015
0.000
0.409--0.013
22
0.404+00.012
0.310+00008$
0.014
0.364+0.050
0.404+00.013
0.226
0.607
0.749
0.626+0.050
0.752-0.0023
0.595+0.061
0.541
0.613+0.061
0.521
0.622+t0.059
arts
0.573+00.036
0.515+00.229
0.588 10003
0.502+0.061
0,597+00.03
0.551+000335
0.495+0.023
0.006
0.483+0.051
0.576+00.034
0.762+0.0029
0.684
0.097
0.707#0.045
0.805+00.028
education
0.690+00.066
0.637+0.053
0.054
0.649#0.068
0.722+t0.048
0..06
0.62510.0322
0.701:0.03.
0.638
0.709
0.660=10.009
0.437:00.02
0.726
0.465=0.089
0.752+10.015
0.525+00.024
0.375+0.018
0.005
0.418+0.060
0.596+ 0.029
revsubser2-2
12
0.504+0.025
0.365+00022
0.50170.0027
0.401
0.572
16
0.4851
0.35710.016
0..2810.019
0.400
0.549
20
0.473
0.350-L0.013+
0.013
0.000
0.532-L0.017
0.610--0.029
0.348+0.0111
0.600
0.530
4.778--0.031
0.617+0.033
0.356+0.01.3
0.613+0.047
0.546+0.023
0.782--0.031
bibtex
0.614110.07
0.35610.014
0.60910.052
0..0640
0.778
11
0.608=00.000
0.353+0.011
0.603
0.500=0.030
0.772-00037
0.606:10.028 14
0.355+0.009
0.60190.039
0.542#0.022
0.770=1.030
13

---

Table 5: Experimental results of each compared approach (mean-tstd) in terms of Macro-averaging, AUC with different label batch size (denoted hy #lahel). The best and the second best performance of each data methods are highlighted boldface and underline respectively. addition, e/o indicates whether SLAN statistically superior/inferio to the comparing approaches on each data set with pairwise t-test (at 0.05 significance level).
Dataset
#label
LIFT
MUENLPLR
SENCE
LIMIC
SLAN
Macro-averaging AUC (the greater, the better)
0.590+10.029
0.541++0.119
0.595+0..331
0.512+0.016
0.618:0.033
0.579-10.031
0.52610.021
0.58710.033
0.511=0.015
0.602-10.034
0.573=+:0027
0.524+0.021
0.581=0.026
0.509=00004.
0.594=0.034
llog
0.573:1:0000
0.52110.019
0.577110.022
0.509110.013
0.590-10.033
0.570:10.019
0.522::0.017
0.573=0.0223
0.508=00011
0.585-00.33
0.68810.0233
0.589100026
0.69011.0049
0.58010.021
0.725-10.034
0.646=t0.026
0.565+0.017
0.660-0.0336
0.561
0.681-0.033
enron
0.645-10.024
0.56310.014
0.66010.033
0.001
0.679-10.031
12
0.646:0.0022
0.563+0.013
0.659=0.033
0.560=0.014
0.677=0.030
0.70210.022.
0.38510:016
0.73910:028
0.56410.020
0.75810.026
0.647=0.014e
0.561+0.010e
0.601
0.544=0.010
0.687=0.019
recreation
0.641110001
0.55710.011
0.66210.0118
0.542-10.010
0.679-1.011
0.635=10.110
0.555=00010
0.655=0.015
0.538=0.010
0.670=0.015
0.60610,023
0.522
0.6671000225
0.564
0.728
0.601=0.016
0.522+00.09+
0.658
0.556=00011
0.699+0.002
slashdot
0.596.10.012
0.522-00.077
0.65310.0014
0.5031100010
0.692.1.022
0.596:10.012
0.522+0.008
0.653=0.016
0.552=0.011
0.687=0.020
0.697100.02
0.373E0.009
0.705
0.535=00020
0.719-0.016
0.677+0.011
0.567+0.009
0.686
0.531=0.018
0.697-0.013
corelSk
0..611
0.565100000
0.67610.016
0.530:10.017
0.687
0.660+0.015
0.560
0.610
0.529=0.016
0.679=:0.014
22
0.657+0.016
0.559+0.008
0.667+0.017
0.528+0.015
0.676+-0.015
0.731=0.022
0.614#0.015
0.002
0.585=0.030+
0.782-F0.022
0.652-10.023
0.57210.006
0.672210002
0.55410..19
0.684110.022
arts
0.638=0.015
0.504
0.656=0.015
0.549=0.015
0.665=0.011
0.630-F0.014
0.558+00.15
0.645+0.014
0.545+0.015
0.655-000113
0.724+00022
0.592+0.028
0.777+0.022
0.593=0.029
0.792+00002
education
0.62910.013
0.555110.014
0.66710.013
0.552.10.188
0.68110.015
0.623+0.010
0.553+0.014
0.660+0.0122
0.549=0.019
0.674
0.79810.013
0.39610.021
0.839-0.0122
0.53820.027
0.86410.010
0.729+00.022
0.568+00010e
0.769-0.022
0.526+0.020
0.789--0.017
revsubser2-2
12
0.722110019
0.56310.004
0.76310.0118
0.52410.017
0.782
0.711+00015
0.559+0.011
0.750=0.015
0.521=0.015
0.770=
0.705+-0.015
0.555+00011
0.044
0.520+0.015
0.764+
0.718-H0:225
0.529=0001
0.0867
0.593-00017
0.840=00015
10.027
0.52910.012
0..066
0.001
0.005
bibtex
0.706=10.027
0.528+0.0122
0.582
0.590=0.016
0.830=0.0016
17704-0..22
0.527+0.011
0.004
0.589+0.015
0.827+0.016
14
0.703110.02
0.527-0.013
0.583
0.005
0.82710.011
Table 6: Win/tic/loss counts (pairwise t-test 0.05 significant level) for SLAN against other multi-label approaches.
SLAN against
Metries
LIFT
MUENLPLR
SENCE
LIMIC
Ranking loss
27/7/5
39/0/0
16/15/8
29/10/0
One-error
34/5/0
39/0/0
20/19/0
34/5/0
Coverage
28/6/5
39/0/D
18/15/6
29/10/0
Average precision
30/9/0
39/0/0
20/13/6
34/5/0
Macroaveraging AUC
35/4/0
39/0/D
33/6/0
39/0/0
In Total
154/31/10
195/0/0
107/68/20
165/30/0
14

---

Table 7: Experimental results of each compared approach (mean-tstd) in terms of Coverage with different label batch size (denoted by #label). The best and the second best performance of each data methods highlighted in boldface and underline respectively. addition, e/o indicates whether SLAN is statistically superior/inferio to the comparing approaches on each data set with pairwise t-test (at 0.05 significance level).
Dataset
#label
LIFT
MUENLPLR
SENCE
LIMIC
SLAN
Coverage (the smaller, ler, the betler)
0.332++0.030
0.384+0.0276
0.329
0.340=+0033
0.336++0.028
0.348110.032
0.398:10.027
0.34810.036
0.353-10.054
0.348-0.022
llog
0.354-t:0003
0.401=0.026
0.351=0.033
0.358=0.031
0.354+0.030
0.356:100029
0.403+:00022
0.055110.029
0.361-10.027
0.357:00.027
0.356:10.026
0.404::0.020e
0.056=:00026
0.361=00026
0.361
0.419100053
0.405
0.401
0.442.10066
0.399-10.054
0.375=00031
0.445+0.020e
0.367=0.025
0.391
0.363--0.028
enron
0.368
0.442-L0.010e
1..016
0.005
0.358-10.019
0.366=0.015 12
0.4
0.360=0.009
0.380=0.016
0.359=0.0113
0.282110:119
0.36710.021
0.26340.022
0.013-00025
0.261
0.303+0.017
0.378+0.0169
0.290
0.322=0.022+
0.292+000023
recreation
0.30411.014
0.37910.014
0.204
0.322-10.020
0.29510.0018
0.306=10.015
0.379+0.014
0.206
0.320=0.019
0.300=0.016
0.29610.002
0.374
0.26470.027
0.303
0.244:0.029
0.303++00022
0.377+0.018
0.273+0.021
0.312+0.028
0.250+000023
slashdot
0.312110018
0.381200012+
0.001
0.322.10023
0.002
0.311+000200
0.38020.011
0.200
0.322+:0022
0.257
0.310
0.441
0.095
0.372-00037
0.357=0.009
0.316-00.018
0.430+0.015
0.304=0.020
0.375-0.044
0.361+0.013
corelSk
0.022.0.000lo
0.401
0.003
0.37941.008
0.006
0.330::00222
0.435+0.018
0.025
0.383=0.051
0.306
22
0.0234
0.434+0.0155
0.025
0.384+0.051
0.375+000016
0.234
0.35
0.009
0.339
0.242-F00028
0.35110.045
0.39810.0338
0.0094
0.412.10.065
0.325.0..04
arts
0.370=0.027
0.411=0.026
0.359+0.029
0.431=0.0644
0.343=0.029
0.379+-0.030
0.420+0.003
0.368+0.0336
0.036
0.436+0.059
0.351+0.031
0.256
0.324
0.232+0.029
0.297=0.047
0.225
education
0.31110,051
0.35810.005
0.2031
0.34610.066
0.28610.044
0.318+0.034
0.366
0.301
0.355=0.041
0.294=0.032
0.204110.011
0.361
0.001
0.320
0.175-0.005
0.310++0.038
0.419+0.029
002016
0.377+0.044
0.077-0.027
revsubset2-2
12
0.323110.033
0.42510.0025
0.29510.030
0.38510.047
1.0088
16
0.335:10.022
0.428+0.016
0.307+0.020
0.393=0.034
0.300=00001
0.351+00.021
0.439+0.015
0.023
0.404+0.032
4.113+0.019
U.240:
0.250
0.286-000188
0.140=0.023
0.236.10.035
0.3961L0.015
0..05
00.0240
0.139-0.0021
bibtex
0.237+00.00
0.396+0.02
0.258
0.282=0.036
0.142-00.023
0.244400.08
0.401+0.020
0.202+
0.290
2.149-000024
14
0.247100.03
0.402-10.014
0.205
0.293110022
0.151
Table 8: Win/tie/loss counts (pairwise t-test at 0.05 significant level) for SLAN against other anomaly detection approaches.
SLAN against
Metrics
OC-SVM
IFOREST
MUENLFOREST
F-measure
26/4/0
26/4/0
23/7/0
15

---

Table 9: Experimental results of each compared approach (mean-tstd) in terms of One-error with different label batch size (denoted by #label). The best and the second best performance of each data methods highlighted in boldface and underline respectively. addition, e/o indicates whether SLAN is statistically superior/inferior to the comparing approaches on each data set with pairwise t-test (at 0.05 significance level).
Dataset
#label
LIFT
MUENLPLR
SENCE
LIMIC
SLAN
One-error (the smaller, the better)
0.708+00.040
0.766+t0.034
0.657=00.04
0.729=0.041
0.702-+0.057
0.7241100339
0.781
0.705-10.000
0.740-100036
0.71410.0500
llog
0.730=+0.039
0.785+0.033
0.715=00.046
0.747-00033
0.722=00005
0.733:1.0044
0.794:10033l
0.716.10.046
0.745-10.026
0.725
0.734100035
0.797::0.029+
0.717=00000
0.745:10002
0.729-20.043
0.230:10.005
0.29310.136
0.210100009
0.276-10146
0.205-10.093
0.393+00.09
0.436+0.090e
0.376+0.076
0.444+0.101
0.370-00.069
enron
0.436:10.082
0.479:10.077
0.416
0.48410.086
0.413-0.076
12
0.44610.044.
0.4
0.426=0.051
0.493=0.054
0.423-0.048
0.46910.031
0.6121000300
0.413
0.577=00035
0.411
0.501=00020
0.629=0.027
0.499
0.592=0.029
0,460+0.032
recreation
0.503-10.026
0.63310.022
0.467
0.592.10.04
0,47011.0030
0.512+00027
0.636:10.033
0.475
0.599=0.039
0,479=0.028
0.602
0.76210.028
0.587 .
0.66810.040
0.52410.047
0.642+00.03
0.768
0.601
0.674=0.038
0.543+00.039
slashdot
0.657110.032
0.77410.021
0.617100.023
0.69110.035
0.557-0.036
0.657+10.333
0.775+0.020
0.618=0.0244
0.694-0.028
0.562=00.03
0.652:10.014
0.081
0.634
0.732.500224
0.628-0.014
0.706=10.018
0.808+0.016
0.687=0.022
0.756=0.030
0.678+0.025
corelSk
0.719100012
0.809
0.70510.018+
0.058
0.694-10.017
0.725=0.011
0.812:0.011
0.711
0.755=0.017
0.702=:0.013
22
0.735+00.01
0.817+0.009
0.718
0.765+0.013
0.709+
0.393+0.033
0.550+0.049
0.045
0.531
0.347
0.556_10.076
0.64810..000
0.502
0.65910.088
0.52810.076
arts
0.583+0.043
0.664
0.563=00004
0.681=0.064
0.560=0.003
0.616+00.04
0.692+00026
0.50540.040
0.707+0.053
0.592-+0.035
0.372::0.43
0.482+00049
0.313+0.041
0.469-0.067
0.301=0.040
education
0.48010.077
0.54910.071
0.440
0.54410.085
0.428110.067
0,498+0.049
0.568
0.460+0.046
0.557=0.056
0.449
0,435110:116
0.69410
0.350
0.68610.091
0.311
0.585+00008
0.758+0.023
0.516+0.024
0.731+0.081
0.482++0.028
revsubset2-2
12
0.60910.021
0.767-0.0225
00.04
0.73910.077
0.511-1..025
0.632:0.017
0.776+0.023
0.501
0.752=0.074
0.537=0.021
0.640--0.012
0.7817
0.084
0.753+0.071
0.055+0.014
0.832+00066
0.406-
0.611 +0.037
0.281=0.039
0.48910..039
0.82410.018
0..0770.047
0.58930.033
0.274-1..039
bibtex
0.491=0.038
0.824+0.016e
0.472=0.054
0.585=00037
0.279=:0.042
0.499+00.039
0.827+0.011
0.481 +0.056
0.592+0.032
1.286--0.044
14
0.500
0.82410.013
0.48210.036
0.589-0.025
0.287
A.2 Data and Code Availability
Our source code can be found at htttss//paam...... seu, edu. wt/.cn////haa.. /hnzhhagg paper are public, and can be found in Section 4.1
The datasets used in this
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We claim the paper's contributions and scope in the introduction (Section
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discussed the limitations of this paper in the conclusion (Section 5
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Section 4.1 and A. of this paper.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The paper provides the data and code availability in Section A.2
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer Section 4.1 and A.1 of this paper.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In Section and A.I of this paper, we presented the mean and standard deviation of multiple training rounds and conduct pairwise t-test between the proposed approach and the comparing approaches.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Please refer Section 41
9. Code Of Ethics
17

---

Question: Does the research conducted in the paper conform, in every respect. with the NeurIPS Code of Ethics http://tss..pappp.../o//a/a///aa
Answer: [Yes]
Justification: We have already reviewed the NeurlPS Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is foundational research and does not involve societal impact.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not pose such risk.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
18


Reviews:
[{"review_id": "OGzuNeP5Oq", "reviewer": "Reviewer_zY3w", "summary": "The abstract discusses multi-label learning where instances can have multiple labels simultaneously. Traditional approaches assume a closed set scenario where test data labels are predefined during training. However, in real-world situations, new labels can emerge during testing, creating an open and dynamic environment. The paper explores Multi-Label Open Set Recognition (MLOSR), which tackles the challenge of recognizing instances with unknown labels in multi-label settings. The proposed approach, SLAN, utilizes sub-labeling information enriched by structural features in the data space to predict unknown labels. Experimental results across different datasets demonstrate the effectiveness of SLAN in addressing the MLOSR problem.", "questions": "How does the SLAN algorithm perform on datasets with highly imbalanced label distributions?\nDoes the parameter sensitivity analysis mentioned in the paper consider the characteristics of different types of datasets?\nHow adaptable and efficient is the SLAN algorithm for real-time or dynamically changing data streams in practical applications?\nDoes the paper consider the interpretability of the algorithm, i.e., how to explain the predictive results of SLAN?\nFor future work, are the authors planning to combine the SLAN algorithm with deep learning models to enhance the quality of feature representation and the performance of the algorithm?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.  It shows comprehensive analyses of the motivation of the studied problem and the proposed method.\n2.  The paper introduces a novel learning framework called Multi-Label Open Set Recognition (MLOSR), which provides a new direction for multi-label learning research. To solve the MLOSR problem, an approach named SLAN that utilizes sub-labeling information and holistic supervision to recognize unknown labels is proposed. The techniques of SLAN are sound and well-motivated,  offering an effective solution for open set recognition in a multi-label environment. \n3. The paper conducts comprehensive empirical validations on various datasets, showing that the proposed method achieves superior or at least comparable performance across multiple evaluation metrics， which enhances the credibility of the research. The paper includes ah findings and indicates its robustness in multi-label learning.  Extensive sensitivity analysis of the trade-off parameters in the SLAN algorithm are conducted, guiding parameter selection in practical applications.", "weaknesses": "Generalization Ability: The SLAN algorithm may have limited generalization ability when dealing with extremely multi-label datasets, restricting its application in broader scenarios.\nFeature Representation Limitations: Operating within the multi-label learning framework, SLAN might be constrained by less informative or discriminative feature representations.\nComputational Resources: While the paper mentions the computational resources for the experiments, it does not elaborate on the computational efficiency and scalability of the algorithm, which may impact its application to large-scale datasets.\nTheoretical Foundation: The paper does not provide theoretical results or proofs, which might reduce the depth of understanding of the algorithm's performance.\nAccess to Code and Datasets: Although the datasets are public, the paper states that the code will be released after acceptance, which may temporarily limit the reproducibility of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. How does the SLAN algorithm perform on datasets with highly imbalanced label distributions?\n\n- **Response 1:** Thanks to the comment. The following table summarizes the level of class-imbalance on data sets employed in the experiments including the minimum, maximum and average imbalance ratio across the label space, which shows these data sets are highly imbalanced. Macro-averaging AUC is the mostly-used evaluation metric under class-imbalance scenarios. Table 4 in appendix in terms of macro-averaging AUC shows that SLAN achieves superior or at least comparable performance against the comparing approaches.\n|Dataset|min|max|avg| \n|---|---|---|---|\n  |  llog | 6.064 | 37.968 | 20.826 |\n  |  enron | 1.009 | 43.789 | 16.148 |\n  |  recreation | 4.107 | 47.077 | 14.308 |\n  |  slashdot | 5.265 | 34.524 | 15.131 |\n  |  corel5k | 3.464 | 49.000 | 29.401 |\n  |  arts | 3.055 | 45.296 | 12.770 |\n  |  education | 2.173 | 41.017 | 12.109 |\n  |  rcvsubset2-2 | 3.216 | 47.780 | 26.370 |\n  |  bibtex | 6.097 | 49.306 | 32.877 |\n\n---\n\n2. Does the parameter sensitivity analysis mentioned in the paper consider the characteristics of different types of datasets?\n\n- **Response 2:**  Thanks to the comment. The following table illustrates how the performance of SLAN changes with varying parameter configurations on data set llog. The performance of SLAN on both llog and enron exhibits similar variation trends. Thus, for fair comparison, we employ the default parameter setting across all data sets.\n| | | | | | | |\n|---|---|---|---|---|---|---|\n|$\\alpha=$ | 0.001 | 0.01 | 0.1 | 1 | 10 |100|\n|_Ranking loss_ | 0.3600  |  0.3600  |  0.3601  |  0.3590  |  0.3596  |  0.3599|\n|_F-measure_ |0.5976  |  0.6106  |  0.6855  |  0.6725  |  0.6411  |  0.6823|\n|$\\beta=$ | 0.001 | 0.01 | 0.1 | 1 | 10 |100|\n|_Ranking loss_ | 0.3599  |  0.3598   | 0.3602  |  0.3590   | 0.3569 |   0.3568|\n|_F-measure_  |  0.6859  |  0.6859  |  0.6859  |  0.6725 |   0.3549|    0.2712 |\n|$\\gamma=$ | 0.001 | 0.01 | 0.1 | 1 | 10 |100|\n|_Ranking loss_ | 0.3489   | 0.3461 |   0.3437  |  0.3490 |   0.3590  |  0.3600|\n|_F-measure_  |   0.1202   | 0.2554   | 0.1767 |   0.6443 |   0.6725 |   0.6859|\n\n---\n\n3. How adaptable and efficient is the SLAN algorithm for real-time or dynamically changing data streams in practical applications?\n\n- **Response 3:** Thanks to the comment. In order to achieve a faster convergence and a good result, we adopt a warm start strategy.  \n\n  (1) When an unseen instance $\\boldsymbol{x}_u$ appears, parameters can be updated through Algorithm 1 in the Rebuttal PDF file. \n\n  (2) When the prior knowledge indicates that the newly appeared instances associated with the same unknown label, these instances can be added to buffer $B$. Once buffer $B$ is full, a pseudo label assignment $\\boldsymbol{p}$ for the unknown label $l_{q+1}$ will be induced, which denotes whether an instance is associated with such unknown label. A new classifier for $l_{q+1}$ will be constructed, and parameters will be updated through Algorithm 2 in the Rebuttal PDF file.\n\n  (3) These updated or augmented parameters can all be considered as a warm start for the subsequent joint optimization procedure, rather than a complete retraining.\n\n---\n4. Does the paper consider the interpretability of the algorithm, i.e., how to explain the predictive results of SLAN?\n\n- **Response 4:** Thanks to the comment. Intuitively, for multi-label classifier, the learned weights $W$ transparently indicate the importance of each input variable for each class label. Additionally, to achieve better performance of the predictive model, a kernel extension is further utilized for the general nonlinear case. For open set recognizer, the structural information might not be maintained in the sub-label space. The enriched sub-labeling information for an instance with unknown labels is differentiated from labeling information with holistic supervision. Such difference can serve as a criterion to distinguish whether an instance is associated with unknown class labels. In the future, we will conduct further interpretability study for the proposed method.\n\n---\n\n5. For future work, are the authors planning to combine the SLAN algorithm with deep learning models to enhance the quality of feature representation and the performance of the algorithm?\n\n- **Response 5:** Thanks to the comment. In the future, it is promising to design MLOSR approaches with deep learning models. For example, label-specific features will be extracted to for informative and discriminative feature representations generation.\n\n---\n\n6. Generalization Ability: The SLAN algorithm may have limited generalization ability when dealing with extremely multi-label datasets, restricting its application in broader scenarios.\n\n- **Response 6:** Thanks to the comment. Let $q$, $d$ and $m$ denote the number of labels, number of training instances and dimension of input space. The training complexity of one iteration in alternative optimization is $\\mathcal{O}(qm^2+qmd+m^2d)$. It is noteworthy that SLAN learns multiple sub-labeling information matrices of which the number equals $q$, which may be slow when applied to date sets with a large number of labels. This is inevitable if considering sub-labeling information. We will leave efficiency improvement for future work.\n\n---\n\n7. Theoretical Foundation: The paper does not provide theoretical results or proofs, which might reduce the depth of understanding of the algorithm's performance.\n\n- **Response 7:** Thanks to the suggestion. In the future, we will perform detailed theoretical analyses for the proposed method.\n\n---\n\n8. Access to Code and Datasets: Although the datasets are public, the paper states that the code will be released after acceptance, which may temporarily limit the reproducibility of the results.\n\n- **Response 8:** Thanks to the comment. The code for this paper will be released  and the results can be reproduced after the paper is accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ztoqzhXXFG", "reviewer": "Reviewer_vnjs", "summary": "This article introduces a new problem in multi-label open set recognition (MLOSR) and proposes a novel approach named Sub-Labeling Information Reconstruction for MLOSR (SLAN). SLAN utilizes sub-labeling information enriched by structural details in the feature space. Experimental results across various datasets demonstrate its effectiveness in addressing this new challenge of MLOSR.", "questions": "1. There is concern regarding the robustness of the method. How stable is its performance when errors or irrelevant labels appear in the dataset?", "limitations": "NA", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This article focuses on a novel problem within multi-label open set recognition (MLOSR), expanding beyond traditional multi-label learning by tackling the identification of unknown open labels. This problem is particularly relevant in practical applications due to the dynamic and open nature of real-world environments.\n2. The strength of the paper lies in its clear and thorough analysis and description of the SLAN method.\n3. The analysis of SLAN across various experimental metrics on different datasets is comprehensive, providing a robust demonstration of its effectiveness in the experiments.\n4. The paper's process and structure are concise, making it relatively easy to understand.", "weaknesses": "1. In the Parameter Sensitivity Analysis section, there is some explanation regarding the experimental parameter settings, but it primarily involves varying each parameter individually without further elaboration on their combined effects or interactions. It would be beneficial to include more extensive experimental analysis that considers the interactions between different parameter combinations.\n2. The paper asserts that OC-SVM, IFOREST, and MUENLFOREST do not exhibit performance on par with SLAN. It would be advantageous to provide further elaboration on why the parameters of these comparison methods, particularly MUENLFOREST, are configured as they are.  \n3. Is there any insight into how SLAN performs differently compared to other methods on different datasets? Could this difference be related to dataset characteristics and distributions?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. There is concern regarding the robustness of the method. How stable is its performance when errors or irrelevant labels appear in the dataset?\n\n- **Response 1:** Thanks to the comment. The detailed experimental results in terms of _ranking loss_ and _F-measure_ under different error rates are reported in the follwing table. As mitigating errors is not considered in SLAN, the performance of SLAN exhibits a slight degradation when the error rate increases. For multi-label classification, the presence of errors or irrelevant labels will hinder the multi-label classifier to accurately induce decision boundary. Similarly, for open-set recognition, such errors or irrelevant labels will impair the discrimination between sub-labeling information and labeling information with holistic supervision. For future work, identifying and mitigating these errors or irrelevant labels will be considered.\n|   |    |   | Ranking loss |  |    |    |   | F-measure |  | \n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|   |   |   | Error rate |  |    |   |   | Error rate |  | \n|Dataset| #label |  0 | 0.1 | 0.3 | Dataset| #label |  0 | 0.1 | 0.3 | \n|enron| 6 | 0.169$\\pm$0.011 | 0.182$\\pm$0.012 | 0.208$\\pm$0.016 |enron| 6  | 0.406$\\pm$0.096  | 0.381$\\pm$0.085  | 0.375$\\pm$0.087  |\n|| 9 | 0.172$\\pm$0.012 | 0.186$\\pm$0.015 | 0.213$\\pm$0.019 |  | 9  | 0.321$\\pm$0.095  | 0.300$\\pm$0.081  | 0.294$\\pm$0.079  |\n|| 12 | 0.174$\\pm$0.010 | 0.188$\\pm$0.013 | 0.214$\\pm$0.016 | | 12  | 0.281$\\pm$0.098  | 0.262$\\pm$0.084  | 0.256$\\pm$0.081  |\n|slashdot|  3 | 0.260$\\pm$0.022 | 0.296$\\pm$0.022 | 0.340$\\pm$0.021 |  slashdot | 3  | 0.528$\\pm$0.127  | 0.526$\\pm$0.123  | 0.516$\\pm$0.119  |\n|| 5 | 0.268$\\pm$0.019 | 0.304$\\pm$0.018 | 0.348$\\pm$0.020 |  | 5  | 0.428$\\pm$0.105  | 0.427$\\pm$0.104  | 0.419$\\pm$0.101  |\n||  7 | 0.270$\\pm$0.018 | 0.305$\\pm$0.015 | 0.350$\\pm$0.016 | | 7  | 0.350$\\pm$0.069  | 0.350$\\pm$0.068  | 0.344$\\pm$0.067  |\n|corel5k| 7 | 0.266$\\pm$0.013 | 0.310$\\pm$0.013 | 0.366$\\pm$0.011 | corel5k | 7  | 0.653$\\pm$0.015  | 0.635$\\pm$0.012  | 0.568$\\pm$0.013  |\n||  12 | 0.276$\\pm$0.012 | 0.318$\\pm$0.011 | 0.373$\\pm$0.010 |  | 12  | 0.539$\\pm$0.017  | 0.527$\\pm$0.015  | 0.482$\\pm$0.016  |\n||  17 | 0.283$\\pm$0.012 | 0.324$\\pm$0.010 | 0.376$\\pm$0.009 |  | 17  | 0.462$\\pm$0.012  | 0.454$\\pm$0.013  | 0.420$\\pm$0.014  |\n||  22 | 0.286$\\pm$0.013 | 0.327$\\pm$0.009 | 0.378$\\pm$0.008 | | 22  | 0.414$\\pm$0.015  | 0.408$\\pm$0.016  | 0.381$\\pm$0.016  |\n\n---\n\n2. In the Parameter Sensitivity Analysis section, there is some explanation regarding the experimental parameter settings, but it primarily involves varying each parameter individually without further elaboration on their combined effects or interactions. It would be beneficial to include more extensive experimental analysis that considers the interactions between different parameter combinations.\n\n- **Response 2:** Thanks to the suggestion. The following table illustrates how the performance of SLAN changes with varying $\\gamma$ and $\\beta$ on data set enron. SLAN achieves relatively stable performance on _ranking loss_ and somewhat sensitive on _F-measure_, whose trend is similar to when only one parameter is changed. We will elaborate on more different parameter combinations in the revised version.\n||||  Ranking  |  loss ||| |  |   |    |  F-measure |  |  |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|   |   |   |   | $\\beta$  |  |  |   |   |   |   | $\\beta$  |  |  |\n||| 0.001 | 0.01  | 0.1   | 1     | 10     |    |    | 0.001 | 0.01  | 0.1   | 1     | 10     |\n|| 0.001 | 0.1734| 0.1730| 0.1698| 0.1802| 0.1908 |     | 0.001 | 0.1701 | 0.0618 | 0.1300 | 0.0749 | 0.0859 |\n|| 0.01  | 0.1736| 0.1724| 0.1701| 0.1757| 0.1828 |      | 0.01  | 0.1808 | 0.0964 | 0.1771 | 0.0742 | 0.0820 |\n| $\\gamma$   | 0.1   | 0.1707| 0.1703| 0.1686| 0.1674| 0.1684 |  $\\gamma$    | 0.1   | 0.2989 | 0.2783 | 0.2408 | 0.2094 | 0.2248 |\n|    | 1     | 0.1726| 0.1726| 0.1719| 0.1714| 0.1711 |      | 1     | 0.3033 | 0.3011 | 0.2812 | 0.2636 | 0.2472 |\n|    | 10    | 0.1742| 0.1741| 0.1741| 0.1740| 0.1740 |      | 10    | 0.3030 | 0.3023 | 0.2929 | 0.2313 | 0.2113 |\n\n---\n\n3.  The paper asserts that OC-SVM, IFOREST, and MUENLFOREST do not exhibit performance on par with SLAN. It would be advantageous to provide further elaboration on why the parameters of these comparison methods, particularly MUENLFOREST, are configured as they are.\n\n- **Response 3:**  Thanks to the suggestion. For OC-SVM, IFOREST, and MUENLFOREST, parameter configurations suggested in respective literatures [1,2] are employed. The following table illustrates how the performance of MUENLFOREST changes with varying parameter configurations on data set enron. The results show that the MUENLFOREST approach is not very sensitive to the settings of parameters.\n||||||\n|:-:|:-:|:-:|:-:|:-:|\n|$q=$|1|3|5|7|\n|F-measure|0.3158|0.3137|0.3179|0.3165|\n|$e_m=$|7|8|9|10|\n|F-measure|0.3178|0.3151|0.3124|0.3110|\n|$C_1=$|0.001|0.01|0.1|1|\n|F-measure|0.3130|0.3135|0.3143|0.3162|\n|$C_2=$|0.001|0.01|0.1|1|\n|F-measure|0.3115|0.3188|0.3154\t|0.3103|\n\n  [1] X.-R. Yu, D.-B. Wang, M.-L. Zhang. Partial label learning with emerging new labels. Machine Learning, 2024, 113(4): 1549-1565.\n\n  [2] Y. Zhu, K.-M. Ting, Z.-H Zhou. Multi-label learning with emerging new labels.  IEEE Transactions on Knowledge and Data Engineering, 2018, 30(10):1901–1914.\n\n---\n\n4. Is there any insight into how SLAN performs differently compared to other methods on different datasets? Could this difference be related to dataset characteristics and distributions?\n\n- **Response 4:**  Thanks to the comments. For multi-label classification and open-set recognition, SLAN achieves superior or at least comparable performance against the comparing approaches in most cases. Generally speaking, the increase in number of labels and label density helps differentiating the sub-labeling information from holistic supervision, facilitating the learning process of classifier and recognizer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yfR8M1ZE8p", "reviewer": "Reviewer_wpso", "summary": "This paper addresses the problem of classifying instances with unknown labels in a multi-label setting (multi-label open set recognition). A\nnovel approach named SLAN is proposed which leverages sub-labeling information, and unknown labels are recognized by differentiating the sub-labeling information from holistic supervision. The effectiveness of the proposed method is validated on different benchmarks.", "questions": "- There are existing tasks with similar settings. For example, in image tasks, there are zero-shot multi-label classification, and open vocabulary multi-label classification. What's the difference between the proposed task and these tasks? Is the setting more applicable?\n\n- It would be better if the paper can include state-of-the-art methods for zero-shot and open-vocabulary multi-label classification in the experimental comparison.\n\n- It would be better to include a flowchart outlining the key step in the proposed approach, which can improve the clarity and make it easier to follow.", "limitations": "The limitation and societal impact is addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper presents a detailed and well-formulated methodology, including the method for structural information discovery and the part related to optimization.\n\n- Extensive experiments on various datasets are conducted to prove the effectiveness of the proposed SLAN approach. The empirical results demonstrate superior or comparable performance to the mentioned methods.", "weaknesses": "- The paper introduces the MLOSR problem, which is novel, but it may overlap with existing tasks like zero-shot multi-label classification and open-vocabulary multi-label classification.\n\n- This paper has limited baselines for comparison. While the paper includes comparisons with several multi-label learning and anomaly detection approaches, it should also compare with state-of-the-art methods specifically designed for zero-shot and open-vocabulary multi-label classification.\n\n- The proposed SLAN approach need to solve multiple optimization problems, which may not scale well to large datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. There are existing tasks with similar settings. For example, in image tasks, there are zero-shot multi-label classification, and open vocabulary multi-label classification. What's the difference between the proposed task and these tasks? Is the setting more applicable?\n\n- **Response 1:** Thanks to the comment. Although multi-label open set recognition (MLOSR), multi-label zero-shot classification (MLZSL) and open vocabulary multi-label classification (OVML) share similar goals, there are essential differences between them. During training phase, in addition to the training set, MLZSL and OVML needs additional prior knowledge of unseen labels ,whereas MLOSR does not. Specifically, MLOSR solely trains on the training set consist of instance-labels pairs. MLZSL additionally needs to extract the semantic information from the observed label space, that is, the relevant label embeddings and the relation between seen and unseen label embeddings. In OVML, visual-related language data like image captions can serve as auxiliary supervision , which potentially has an implicit intersection with the unseen labels. During testing, due to the lack of prior knowledge, MLOSR only needs to identify novel labels and mark them as unknown, while MLZSL and OVML must classify unknown labels into specific labels. \n\n  MLOSR can be considered as an extreme case of MLZSL and OVML. Each of these tasks has its own applicable real-world scenarios. It is necessary to determine the specific task based on the availability of prior knowledge.\n\n---\n\n2. It would be better if the paper can include state-of-the-art methods for zero-shot and open-vocabulary multi-label classification in the experimental comparison.\n\n- **Response 2:**  Thanks to the suggestion. As mentioned above, MLOSR is differentiated from MLZSL and OVML based on whether there is prior knowledge of unseen labels or not. Thus, it is inappropriate to apply approaches designed for MLOSR to MLZSL and OVML scenarios without requisite prior knowledge, and vice versa.\n\n---\n\n3. It would be better to include a flowchart outlining the key step in the proposed approach, which can improve the clarity and make it easier to follow.\n\n- **Response 3:** Thanks to the suggestion. The flowchart of SLAN is summarized in Figure 1 in the Rebuttal PDF file. Given the multi-label training set, a weighted graph is constructed to characterize the manifold structure of feature space. After that, the alternative optimization strategy is adopted to optimize open set recognizer and multi-label classifier simultaneously.\n---\n\n4. The paper introduces the MLOSR problem, which is novel, but it may overlap with existing tasks like zero-shot multi-label classification and open-vocabulary multi-label classification.\n\n- **Response 4:** Please kindly refer to **Response 1**.\n---\n\n5. This paper has limited baselines for comparison. While the paper includes comparisons with several multi-label learning and anomaly detection approaches, it should also compare with state-of-the-art methods specifically designed for zero-shot and open-vocabulary multi-label classification.\n\n- **Response 5:** Please kindly refer to **Response 2**.\n\n---\n\n6. The proposed SLAN approach need to solve multiple optimization problems, which may not scale well to large datasets.\n\n- **Response 6:** Let $q$, $d$ and $m$ denote the number of labels, number of training instances and dimension of input space. Our optimization algorithm mainly contains several steps. Before alternative optimization, we need to instantiate $\\mathbf{S}$ with ADMM in $\\mathcal{O}(m^3)$. Then our method needs to iteratively solve three optimization problem w.r.t. $\\mathbf{Z},\\mathbf{F}_k,\\mathbf{W}$, which can be solved in $\\mathcal{O}(qmd),\\mathcal{O}(qm^2)$ and $\\mathcal{O}(m^2d)$  respectively. In summary, the overall complexity of our optimization algorithm is the sum of these operations mentioned above. It is noteworthy that the complexity of SLAN is related to $q$ and $m^3$ , which may be slow when applied to date sets with a large number of labels and instances. We will leave efficiency improvement for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We greatly appreciate all of you for your thoughtful comments and valuable suggestions. These are very helpful for improving our paper. We have carefully referred to the questions and written the response. In addition to the text responses, we also report some figure results in the PDF file. We hope the responses would meet your requirements.

Best Regards,

Authors

Meta-review: This paper was reviewed by three experts in the field and received "Weak Accept", "Accept" and "Borderline Accept" as the ratings. The reviewers agreed that the proposed multi-label open set recognition framework introduces a new direction for multi-label learning research, the experimental analysis is comprehensive with encouraging results, and that the paper is well-written and easy to follow. 

The reviewers raised concerns about the robustness of the proposed method, which was addressed convincingly by the authors in the rebuttal. The authors also conducted experiments to study the interactions between different parameter combinations, in response to a comment from one of the reviewers. Concerns were also raised about the parameter sensitivity analysis, which was also addressed convincingly by the authors in the rebuttal.  

The reviewers, in general, have a positive opinion about the paper and its contributions. Based on the reviewers' feedback, the decision is to recommend the paper for acceptance to NeurIPS 2024. The reviewers have provided some valuable comments, such as the theoretical analyses of the proposed method, its computational efficiency when dealing with datasets with a large number of labels and designing multi-label open set recognition algorithms with deep learning models. The authors are encouraged to address these in the final version of their paper. The authors are also encouraged to make their code publicly available for better reproducibility. We congratulate the authors on the acceptance of their paper!
Decision: Accept (poster)