Title: Multi-Label Open Set Recognition
Venue: NeurIPS 2024
Accepted: yes
Keywords: machine learning

multi-label learning

open set recognition
Abstract: In multi-label learning, each training instance is associated with multiple labels simultaneously. Traditional multi-label learning studies primarily focus on closed set scenario, i.e. the class label set of test data is identical to those used in training phase. Nevertheless, in numerous real-world scenarios, the environment is open and dynamic where unknown labels may emerge gradually during testing. In this paper, the problem of multi-label open set recognition (MLOSR) is investigated, which poses significant challenges in classifying and recognizing instances with unknown labels in multi-label setting. To enable open set multi-label prediction, a novel approach named SLAN is proposed by leveraging sub-labeling information enriched by structural information in the feature space. Accordingly, unknown labels are recognized by differentiating the sub-labeling information from holistic supervision. Experimental results on various datasets validate the effectiveness of the proposed approach in dealing with the MLOSR problem.

Multi-Label Open Set Recognition

Abstract

In multi-label learning, each training instance is associated with multiple labels simultaneously. Traditional multi-label learning studies primarily focus on closed set scenario, i.e. the class label set of test data is identical to those used in training phase. Nevertheless, in numerous real-world scenarios, the environment is open and dynamic where unknown labels may emerge gradually during testing. In this paper, the problem of multi-label open set recognition (MLOSR) is investigated, which poses significant challenges in classifying and recognizing instances with unknown labels in multi-label setting. To enable open set multi-label prediction, a novel approach named SLAN is proposed by leveraging sub-labeling information enriched by structural information in the feature space. Accordingly, unknown labels are recognized by differentiating the sub-labeling information from holistic supervision. Experimental results on various datasets validate the effectiveness of the proposed approach in dealing with the MLOSR problem.

1 Introduction

Multi-label Learning (MLL) deals with the problem where an instance can be associated with multiple labels simultaneously [37, 19]. As a practical machine learning paradigm, multi-label learning has been widely applied in various real-world applications, such as image annotation [30], text categorization [26], information retrieval [12].

Traditional multi-label learning studies focus on closed set scenario. That is, they assume that the class label set of test data is identical to that in the training set [37, 8, 21]. However, in many real-world scenarios, this assumption rarely holds because the environment is open and dynamic [29]. In addition to the extant label knowledge at training phase, the unknown labels may emerge gradually with the data streams during the testing phase. For example, in Figure 1, the test image is annotated with several relevant labels, some of which are unseen in the training set. The classification task becomes much more challenging because the label correlation between known and unknown labels may degrade the performance of the predictive model. Furthermore, due to the presence of unknown labels in the class label set of test data, these test data are hardly employed in subsequent learning processes, such as incremental learning [15].

Motivated by the potential applications, we formalize a novel framework named multi-label open set recognition (MLOSR), whose goal is to learn a multi-label model that can correctly classify known labels for the unseen instance and recognize unknown labels within its relevant label set. This can be regarded as a special weakly supervised learning. In MLOSR, the most challenging part is to recognize the unknown labels associated with instances. Since we do not have any prior knowledge of unknown labels and they almost always co-occur with some known labels, it is very difficult to separate instances with unknown labels from those with the known labels only.

The relevant label set

cloud

car

ship

sea

mountain

building

tree

sky

trestle bridge

…

Open set recognition (OSR) is a paradigm previously proposed in [24] and is formalized as a riskminimizing constrained functional optimization problem. OSR describes a scenario where new classes unseen in training occur in testing, thus classifiers must be able to properly identify seen samples while rejecting unseen ones [9]. [24] proposes a 1-vs-Set machine to minimize open set risk by sculpting a decision space from the marginal distances of binary SVM. OSR problem is further studied via algorithm adaptation [1, 16], statistical extreme value theory (EVT) [25, 32], margin distribution [23] or hierarchical Dirichlet process [10]. These existing works are based on the fact that each instance owns one ground-truth label for multi-class cases. Thus, they cannot be used to directly solve the MLOSR problem, due to the multiple ground-truth labels in MLL.

To address MLOSR problem, we propose a tailored algorithm named SLAN, i.e. Sub-Labeling informAtion reconstructioN for multi-label open set recognition . The basic strategy of SLAN is to enrich the sub-labeling information in the sub-label space by leveraging the structural information in the feature space and differentiating it from the labeling information from holistic supervision. Specifically, the underlying structure of feature space is characterized by the sparse reconstruction relationships among training instances. After that, the reconstruction information is utilized to guide the enrichment of sub-labeling information. Then, a unified optimization framework is presented to simultaneously facilitate open set recognizer and multi-label classifier induced with alternating optimization. Our empirical study on datasets from diverse domains demonstrates the effectiveness of the proposed approach.

The rest of the paper is organized as follows. We present a brief review of related works. Then we formulate the problem and propose the algorithm. Next, experimental results are reported, followed by the conclusion.

2 Related Work

The task of multi-label learning has been extensively studied in recent years [37, 19]. Generally, the major challenge for multi-label learning is its huge output space which is exponential to the number of class labels. Therefore, exploiting label correlations has been adopted as a common strategy to facilitate the learning process. Roughly speaking, extant approaches can be grouped into three categories based on the order of correlations, i.e. first-order approaches, second-order approaches and high-order approaches. First-order approaches tackle multi-label learning problem in a label-by-label manner [2, 33]. Second-order approaches exploit pairwise interactions among class labels [7, 3]. High-order approaches exploit relationships among a subset of or all class labels [22, 14].

OSR is critical for the tasks where incomplete knowledge exists at training time, and unknown classes can be submitted to an algorithm during the testing phase, requiring the classifiers to classify the seen classes and deal with unseen ones. According to [9], traditional machine learning methods are adapted to OSR scenario. For instance, SVM-based models add extra constraints on the score space in [24, 4]; A collective decision-based model implemented by hierarchical Dirichlet process is proposed in [10]; and distance-based models [1, 16] are developed by modifying existing classifiers, such as nearest class mean classifier and nearest neighbor classifier. Some other approaches focus

on the EVT [17]. [25] combines the EVT for score calibration with two separated compact abating probability SVMs, where the first SVM is used as a conditioner and the second SVM fitted yields the posterior estimate. [32] transforms the OSR problem into a set of hypothesis testing problems by modeling the tail part of reconstruction error distribution via EVT. [23] formulates the extreme value machine with distributional information, which can be interpreted by EVT. There are also some methods trying to incorporate few-shot learning into OSR [5].

MLOSR can be regarded as a combination of MLL and OSR. Thus, a straightforward approach is to generate a independent recognizer besides the multi-label classifier. However, as unknown labels may co-occur with known labels, it is difficult to separate instances with unknown labels from instances with known labels only, which leads to OSR approaches that could not be applied in MLOSR problems.

Streaming multi-label learning (SMLL) [31] is similar to our MLOSR problem but differs in the setting of unknown labels. It aims to derive a unified model by taking care of the continually emerging new unknown labels on the training data. [31] trains a linear classifier for new labels with the linear hypotheses between labels and classifiers. [28] proposes a novel DNN-based framework to model the emerging new labels depending on high-order representations. [29] presents probabilistic streaming label tree to incorporate new labels, which capture hierarchical correlations among labels. Compared to SMLL, MLOSR is much more challenging in recognizing unknown labels as the unknown labels only emerge in testing phase. In the next section, the first attempt towards MLOSR is introduced.

3 The SLAN Approach

3.1 Problem Formulation

Formally, let X = R d denote the d -dimensional input space and Y = { l 1 , l 2 , . . . , l q } denote the label space including q class labels. Each multi-label instance can be denoted as ( x i , Y i ) , where x i ∈ X is its feature vector and Y i ⊆ Y is the set of relevant labels associated with x i . Here, a q -dimensional indicator vector y i = [ y i 1 , y i 2 , ...y iq ] ⊤ is utilized to denote Y i , where y ik = 1 indicates class label l k ∈ Y and y ik = -1 otherwise. By arranging feature vectors and label vectors of m training instances, we obtain the feature matrix X = [ x 1 , . . . , x m ] and label matrix Y = [ y 1 , . . . , y m ] .

Given the multi-label training set D = { ( x i , Y i ) | 1 ≤ i ≤ m } , the goal of MLOSR is to learn a model from D that can correctly classify known labels for the unseen instance and recognize unknown labels within its relevant label set. Conceptually, given the multi-label training set D , an open space risk function R O and an empirical risk function R ε , multi-label open set recognition aims to derive a measurable recognition function f ∈ H by minimizing the following Open Set Risk :

where λ r is a regularization constant.

3.2 Structural Information Discovery

To characterize the underlying manifold structure of feature space, a weighted directed graph G = ( V , E , S ) is constructed over the set of training instances, where V = { x i | 0 ≤ i ≤ m } corresponds to the set of vertices and E = { ( x i , x j ) | s ij = 0 , 1 ≤ i = j ≤ m } corresponds to the set of edges from x i to x j with nonzero weight.

̸

̸

Furthermore, S = [ s ij ] m × m corresponds to the weight matrix encoding the relationships among all training instances. Conceptually, the weight value s ij reflects relative importance of x i in reconstructing x j . Thus, by implementing global sparse reconstruction, the weight matrix S is instantiated by solving the following optimization problem:

Here, the first term controls the linear reconstruction error via squared Frobenius norm while the second term controls the sparsity of reconstruction via ℓ 1 norm. The relative importance is balanced

by the trade-off parameter µ 0 . Then, the constrained optimization problem in Eq.(2) can be solved via a standard ADMM (Alternating Direction Method of Multiplier) procedure [11].

3.3 Sub-Labeling Information Enrichment

According to the manifold assumption, the structural relationship specified in the feature space should also be preserved in the entire label space to enrich labeling information originally encoding in the indicator vector y i [13, 38]. That is, y i can be transformed into a numerical labeling vector z i = [ z i 1 , z i 2 , . . . , z iq ] ⊤ under holistic supervision which encodes richer semantics for predictive model induction.

However, this assumption might be suboptimal in the entire label space. Considering one specific label l k ∈ Y , let f k i = [ f k i 1 , . . . , f k i,k -1 , f k i,k +1 , . . . , f k iq ] ⊤ represent the enriched sub-labeling information of x i . With the manifold assumption, the structural information would be maintained in the sub-label space Y\{ l k } . Then, instances with l k can be reconstructed via the instances without l k as well as the enriched sub-labeling information. Nevertheless, these reconstructed instances can not be assigned with l k since there's not enough positive labeling information w.r.t. l k . That is, the sub-labeling information is differentiated from labeling information with holistic supervision. Thus, in open set scenario, if l k is specified as an unknown class label, such difference can be employed as a criterion for one specific instance to distinguish whether it is associated with an unknown class label.

Let concatenate all f k i , denoted by F k = [ f k 1 , . . . , f k m ] ∈ R ( q -1) × m and all z i , denoted by Z = [ z 1 , . . . , z m ] ∈ R q × m . The enriched sub-labeling information is generated via leveraging the structural information encoded in S by solving the following optimization problem:

Here, δ k i is a indicator variable, where δ k i = 1 if l k is not associated with x i ; otherwise δ k i = 0 . P k is a ( q -1) × q projection matrix to align z i with f k i without regard to l k , which removes the k -th row of the identity matrix. The first term supervises the labeling information from the holistic aspect. The second term conveys the manifold structure specified in the feature space to the sub-label space. The third term minimizes the labeling information difference between sub-label space and entire label space to differentiate instances with or without such unknown class label, which implicitly reduces the open space risk.

During the testing phase, for an unseen instance x ∗ , the reconstruction coefficients w.r.t. l k is identified by resorting to the ADMM technique over the training set. After that, the enriched sublabeling information f k ∗ of x ∗ is determined via the second term of Eq.(3). Thereafter, whether unknown labels are associated with x ∗ is determined by the following recognizer G via ensemble majority voting:

where g k ( x ∗ ) = -1 if || f k ∗ -P k z ∗ || 2 2 > ρ k ; otherwise g k ( x ∗ ) = 1 . ρ k is the threshold, and can be chosen so that 100 × τ %instances with l k in training set satisfy || f k i -P k z i || 2 2 > ρ k . The labeling information z ∗ under holistic supervision is generated by the following MLL classifier.

3.4 MLLClassifier Training

Similar to the existing strategy in previous OSR algorithms, instead of training the classifier independently, we perform open set recognizer and multi-label classifier induced simultaneously. Then, the sub-labeling information can be optimized by considering both the manifold assumption and model outputs. Furthermore, considering the label correlation between known and unknown labels, such jointly optimization procedure with recognizer can facilitate the classifier more robust. We denote W ∈ R q × d and b ∈ R q as a multi-label classifier and adopt the squared Frobenius norm as the

regularization term to control the model complexity:

Let B k = [ b k pi ] ( q -1) × m denote the indicator matrix with b k pi = δ k i . Then, the objective function of the unified framework is shown as follows:

Here, The first two terms control the open space risk and remaining terms control the empirical risk.

3.5 Alternative Optimization

Update Z With fixed F 1 , . . . , F q , W and b , the optimization problem Eq.(6) can be stated as follows:

The above optimization problem can be solved by updating Z with gradient descent. The gradient of the objective function w.r.t. Z is

Update F 1 , . . . , F q With Z , W and b fixed, the optimization problem Eq.(6) can be stated as follows:

Similarly, gradient descent is employed, and the gradient w.r.t. F k is:

where T = ( S -I m × m )( S -I m × m ) ⊤ .

Update W and b While Z and F 1 , . . . , F q are fixed, the optimization problem (4) can be stated as follows:

Here, E = [ e 1 , . . . , e m ] ∈ R q × m , where e i = z i -( W x i + b ) . To achieve better performance of the predictive model, a kernel extension is further facilitated for the general nonlinear case. Let Φ = [ ϕ ( x 1 ) , . . . , ϕ ( x m )] ∈ R h × m , where ϕ ( · ) : R d → R h corresponds to the feature mapping that maps the feature space to some higher dimensional Hilbert space with h dimensions. Then, the Lagrangian function of this problem can be formulated as:

Algorithm 1 The pseudo-code of SLAN

Input : The multi-label training set D , the trade-off parameters α, β, γ, µ 1 , τ , an unseen instance x ∗ ; Output : The predicted label set Y ∗ for x ∗ , the recognition result G ( x ∗ ) .

Process :

1: Instantiate the weighted graph G = ( V , E , S ) by solving Eq.(2) with ADMM procedure;

2: Calculate the kernel matrix K = [ κ ( x i , x j )] m × m ;

3: Initialize Z with Y ;

4: Initialize F k with P k Y ( 1 ≤ k ≤ q );

5: repeat

6: Update Z according to Eq.(7);

7: Update F k according to Eq.(9);

8: Update W and b according to Eq.(11);

9: until convergence or maximum number of iterations being reached

10: return Y ∗ and G ( x ∗ ) according to Eq.(14) and Eq.(4).

where A = [ a ki ] ∈ R q × m stores the Lagrange multipliers. According to the KKT conditions, we can obtain:

where H = 1 µ 1 K + I m × m and K = Φ ⊤ Φ with its element k ij = κ ( x i , x j ) = ϕ ( x i ) ⊤ ϕ ( x j ) based on the chosen kernel function κ ( · , · ) . Then, by incorporating the specified kernel function, the modeling output is denoted by 1 µ 1 AK + b1 ⊤ m . Furthermore, given an unseen instance x ∗ ∈ X , its relevant label set is predicted as:

The pseudo-code of SLAN is summarized in Algorithm 1. Given the multi-label training set, a weighted graph is constructed to characterize the manifold structure of feature space (Step 1). After that, the alternative optimization strategy is adopted to optimize open set recognizer and multi-label classifier simultaneously (Step 2-9). Finally, the relevant label set of the unseen instance is predicted and the recognition result is generated based on the learned model (Step 10).

4 Experiments

4.1 Experimental Setup

Table 1 summarizes the detailed characteristics of each benchmark multi-label data set S employed in the experiments, including the number of instances |S| , number of features dim ( S ) , number of class labels L ( S ) , label cardinality LCard ( S ) , label density LDen ( S ) , number of distinct label sets DL ( S ) and proportion of distinct label sets PDL ( S ) . To alleviate the influence of extreme

imbalance, any class label with rare appearance or with overly-high imbalance ratio is excluded from the label space following previous settings [34].

For a given dataset, we randomly select 50% labels as known labels and the remaining labels as unknown labels with different label batch sizes. Then, we sample 60% instances without unknown labels to form training set while the remaining instances are treated as test data. The sampling procedure is repeated ten times, and the mean metric value as well as standard deviation for each label batch are reported.

To evaluate the performance of multi-label classifiers, we utilize five widely-used multi-label evaluation metrics [37], including Ranking loss , One-error , Coverage , Average precision and Macroaveraging AUC . In addition, F-measure is employed to evaluate the performance of recognizers.

To validate the effectiveness of the proposed SLAN approach in multi-label learning, four multi-label learning approaches are used for comparative studies.

LIFT [35]: A multi-label learning approach, which induces classifiers with the label-specific features generated via conducting clustering analysis for each class label. [parameter configuration: r = 0 . 1 ]

MUENLPLR [39]: A SVM-based dynamic multi-label learning approach which trains a set of linear classifiers by minimizing misclassification loss and pairwise ranking loss. [parameter configuration: C 1 = 1 , C 2 = 1 ]

SENCE [27]: A multi-label learning approach based on label-specific features generated by mixture-based clustering ensemble. [parameter configuration: r = 0.4].

LIMIC [21]: A multi-semantics multi-label metric learning approach coupled with MLLNN [36] which learns one global and multiple label-specific local metrics simultaneously. [parameter configuration: λ 1 = 1 , λ 2 = 100 , γ = 2 , K = 10 ].

With the first attempt towards solving the MLOSR problem, there is no method can be directly applied. Thus, we compare the proposed SLAN approach with existing anomaly detection approaches.

OC-SVM [20]: A SVM-based approach which constructs a hyper-sphere surrounding all instances from known labels.

IFOREST [18]: An unsupervised forest-based anomaly detection approach which employs average path length over all trees as the anomaly score.

MUENLFOREST [39]: A forest-based dynamic multi-label learning approach which utilizes clustering process in each nodes by considering the feature space and the label patterns. [parameter configuration: q = 5 , ψ = 256 , g = 100 , e m = 9 ].

For the proposed SLAN approach, trade-off parameters are set as α = 0 . 1 , β = 0 . 1 , γ = 10 , µ 1 = 0 . 1 , τ = 0 . 8 . µ 0 is fixed to be 0.1. The sensitivity analysis of parameter configurations is conducted in Subsection 4.3. A Linux server equipped with Intel Xeon CPU (48 cores @ 2.67GHz) and 256GB memory is used for supporting the experiments.

4.2 Experimental Results

The detailed experimental results in terms of Ranking loss and F-measure are reported in Table 2-3. Due to the page limit, the results on other metrics are shown in the Appendix. Meanwhile, pairwise t-test [6] is conducted to demonstrate whether the performance of SLAN is statistically superior/inferior to the comparing approaches on each data set. The resulting win/tie/loss counts are summarized in the supplementary material.

Based on the reported experimental results, the following observations can be made:

Compared with the performance on close set instances (#label = 0), the performance of all comparing approaches on open set instances degrades. Nonetheless, across all multi-label evaluation metrics, SLAN achieves superior or at least comparable performance against the comparing approaches in 91.7% cases. The results clearly indicate the jointly optimization with recognizer serves a more effective way to achieve more robust multi-label classifier in the open environment.

Comparing with anomaly detection approaches, SLAN achieves better performance in 83.3% cases. Possible reasons are that: (a) OC-SVM and IFOREST are previously designed for multiclass scenario which can not directly solve MLOSR problem. (b) For the construction of MUENLFOREST, instance is augmented with its predictive values derived from MUENLPLR. However, the predictive values might be suboptimal as MUENLPLR is trained without considering open space risk.

In the multi-label setting, instances with unknown labels may share the same dense region of instances with known labels, which makes multi-class anomaly detection approaches tend to reject instances with unknown labels. That is why since MUENLPLR is under multi-label setting, it still inferior to IFOREST and OC-SVM.

4.3 Parameter Sensitivity Analysis

In this section, we study the sensitivity analysis of trade-off parameters α, β, γ, µ 1 , τ shown in Algorithm 1. Figure 2 illustrates how the performance of SLAN changes with varying parameter configurations on data set enron. As shown in Figure 2, SLAN achieves relatively stable performance on multi-label metrics and somewhat sensitive on F-measure . In this paper, trade-off parameters are set as α = 0 . 1 , β = 0 . 1 , γ = 10 , µ 1 = 0 . 1 , τ = 0 . 8 , which can be employed as the default parameter setting.

-3

-2

-1

0

1

2

value of log

10

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

performance

F-measure

Ranking loss

One-error

Covrage

Average precision

Macro-averaging-AUC

(a) Varying

α

-3

-2

-1

0

1

2

value of log

10

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

performance

F-measure

Ranking loss

One-error

Covrage

Average precision

Macro-averaging-AUC

(b) Varying

β

-3

-2

-1

0

1

2

value of log

10

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

performance

F-measure

Ranking loss

One-error

Covrage

Average precision

Macro-averaging-AUC

(c) Varying

γ

-3

-2

-1

0

1

2

value of log

10

1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

performance

F-measure

Ranking loss

One-error

Covrage

Average precision

Macro-averaging-AUC

(d) Varying

µ

1

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

value of

0

0.1

0.2

0.3

0.4

0.5

0.6

F-measure

(e) Varying

τ

5 Conclusion

The major contributions of our work are two-fold: 1) We formalize a novel learning framework named multi-label open set recognition (MLOSR), which aims to classify and recognize instances with unknown labels in multi-label setting, suggesting a new direction for multi-label learning. 2) We propose a novel MLOSR approach named SLAN which can facilitate open set multi-label classification by utilizing sub-labeling information and recognize the unknown labels by differentiating the sublabeling information from holistic supervision. Extensive experimental results clearly validate the effectiveness of the proposed SLAN approach.

However, SLAN enriches #labels+1 sub-labeling information, which could hardly generalize to extreme multi-label data set. Meanwhile, SLAN works in the multi-label learning schema where

the feature representations of instances may less informative. In the future, it is interesting to investigate towards extreme multi-label learning to achieve tolerable scalability and design deep MLOSRapproaches with discriminative feature representations. Furthermore, it is desirable to extend evaluation metrics for MLOSR.

Acknowledgments

The authors wish to thank the anonymous reviewers for their helpful comments and suggestions. This work was supported by the National Science Foundation of China (62225602) and the Big Data Computing Center of Southeast University.

References

Abhijit Bendale and Terrance E. Boult. Towards open world recognition. In Proceeding of 2015 IEEE Conference on Computer Vision and Pattern Recognition , pages 1893-1902, Boston, MA, 2015.

Matthew R. Boutell, Jie bo Luo, Xi-Peng Shen, and Christopher M. Brown. Learning multi-label scene classification. Pattern Recognition , 37(9):1757-1771, 2004.

Christian Brinker, Eneldo Loza Mencía, and Johannes Fürnkranz. Graded multilabel classification by pairwise comparisons. In Proceedings of the 14th IEEE International Conference on Data Mining , pages 731-736, Shenzhen, China, 2014.

Hakan Cevikalp. Best fitting hyperplanes for classification. IEEE Trans. Pattern Anal. Mach. Intell. , 39(6):1076-1088, 2017.

Yongjuan Che, Yuexuan An, and Hui Xue. Boosting few-shot open-set recognition with multirelation margin loss. In Proceedings of the 32th International Joint Conference on Artificial Intelligence , pages 3505-3513, Macao, SAR, China, 2023. ijcai.org.

Janez Demsar. Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. , 7:1-30, 2006.

Johannes Fürnkranz, Eyke Hüllermeier, Eneldo Loza Mencía, and Klaus Brinker. Multilabel classification via calibrated label ranking. Machine Learning , 73(2):133-153, 2008.

Yi Gao, Miao Xu, and Min-Ling Zhang. Unbiased risk estimator to multi-labeled complementary label learning. In Proceedings of the 32th International Joint Conference on Artificial Intelligence , pages 3732-3740, Macao, SAR, China, 2023. ijcai.org.

Chuan-Xing Geng, Sheng-Jun Huang, and Song-Can Chen. Recent advances in open set recognition: A survey. IEEE Trans. Pattern Anal. Mach. Intell. , 43(10):3614-3631, 2021.

Chuanxing Geng and Songcan Chen. Collective decision for open set recognition. IEEE Trans. Knowl. Data Eng. , 34(1):192-204, 2022.

Euhanna Ghadimi, André Teixeira, Iman Shames, and Mikael Johansson. Optimal parameter selection for the alternating direction method of multipliers (ADMM): quadratic problems. IEEE Trans. Autom. Control. , 60(3):644-658, 2015.

Siddharth Gopal and Yiming Yang. Multilabel classification with meta-level features. In Proceeding of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 315-322, Geneva, Switzerland, 2010.

Peng Hou, Xin Geng, and Min-Ling Zhang. Multi-label manifold learning. In Proceedings of the 30th AAAI Conference on Artificial Intelligence , pages 1680-1686, Phoenix, AZ, 2016.

Ming Huang, Fu-Zhen Zhuang, Xiao Zhang, Xiang Ao, Zheng-Yu Niu, Min-Ling Zhang, and Qing He. Supervised representation learning for multi-label classification. Machine Learning , 108(5):747-763, 2019.

Sheng-Jun Huang and Zhi-Hua Zhou. Active query driven by uncertainty and diversity for incremental multi-label learning. In Hui Xiong, George Karypis, Bhavani Thuraisingham, Diane J. Cook, and Xindong Wu, editors, Proceedings of the 13th IEEE International Conference on Data Mining , pages 1079-1084, Dallas,TX, 2013.

Pedro Ribeiro Mendes Júnior, Roberto Medeiros de Souza, Rafael de Oliveira Werneck, Bernardo V. Stein, Daniel V. Pazinato, Waldir R. de Almeida, Otávio A. B. Penatti, Ricardo da Silva Torres, and Anderson Rocha. Nearest neighbors distance ratio open-set classifier. Mach. Learn. , 106(3):359-386, 2017.

Samuel Kotz and Saralees Nadarajah. Extreme value distributions: theory and applications . World Wcientific, 2000.

Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In Proceedings of the 8th IEEE International Conference on Data Mining , pages 413-422, Pisa, Italy, 2008.

Wei-Wei Liu, Hao-Bo Wang, Xiao-Bo Shen, and Ivor W. Tsang. The emerging trends of multilabel learning. IEEE transactions on pattern analysis and machine intelligence , 44(11):79557974, 2021.

Jun-Shui Ma and Simon Perkins. Time-series novelty detection using one-class support vector machines. In Proceedings of the 2003 International Joint Conference on Neural Networks , volume 3, pages 1741-1745, Portland, OR, 2003.

Jun-Xiang Mao, Wei Wang, and Min-Ling Zhang. Label specific multi-semantics metric learning for multi-label classification: Global consideration helps. In Proceedings of the 32th International Joint Conference on Artificial Intelligence , pages 4055-4063, Macao, SAR, China, 2023.

Jesse Read, Bernhard Pfahringer, Geoff Holmes, and Eibe Frank. Classifier chains for multilabel classification. Machine Learning , 85(3):333-359, 2011.

Ethan M. Rudd, Lalit P. Jain, Walter J. Scheirer, and Terrance E. Boult. The extreme value machine. IEEE Trans. Pattern Anal. Mach. Intell. , 40(3):762-768, 2018.

Walter J. Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E. Boult. Toward open set recognition. IEEE Trans. Pattern Anal. Mach. Intell. , 35(7):1757-1772, 2013.

Walter J. Scheirer, Lalit P. Jain, and Terrance E. Boult. Probability models for open set recognition. IEEE Trans. Pattern Anal. Mach. Intell. , 36(11):2317-2324, 2014.

Pingjie Tang, Meng Jiang, Bryan (Ning) Xia, Jed W. Pitera, Jeffrey Welser, and Nitesh V. Chawla. Multi-label patent categorization with non-local attention-based graph convolutional network. In Proceedings of the 34th AAAI conference on artificial intelligence , pages 9024-9031, New York, NY, 2020.

Yi-Bo Wang, Jun-Yi Hang, and Min-Ling Zhang. Stable label-specific features generation for multi-label learning via mixture-based clustering ensemble. IEEE CAA J. Autom. Sinica , 9(7):1248-1261, 2022.

Zhen Wang, Liu Liu, and DaCheng Tao. Deep streaming label learning. In Proceedings of the 37th International Conference on Machine Learning , volume 119, pages 9963-9972, Virtual Event, 2020.

Tong Wei, Jiang-Xin Shi, and Yu-Feng Li. Probabilistic label tree for streaming multi-label learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1801-1811, Virtual Event, Singapore, 2021.

Ren-Chun You, Zhi-Yao Guo, Lei Cui, Xiang Long, Ying-Ze Bao, and Shi-Lei Wen. Crossmodality attention with semantic graph embedding for multi-label classification. In The 34th AAAI Conference on Artificial Intelligence , pages 12709-12716, New York, NY, 2020.

Shan You, Chang Xu, Yun-He Wang, Chao Xu, and Da-Cheng Tao. Streaming label learning for modeling labels on the fly. CoRR , abs/1604.05449, 2016.

He Zhang and Vishal M. Patel. Sparse representation-based open set recognition. IEEE Trans. Pattern Anal. Mach. Intell. , 39(8):1690-1696, 2017.

Min-Ling Zhang, Yu-Kun Li, Xu-Ying Liu, and Xin Geng. Binary relevance for multi-label learning: An overview. Frontiers of Computer Science , 12(2):191-202, 2018.

Min-Ling Zhang, Yu-Kun Li, Hao Yang, and Xu-Ying Liu. Towards class-imbalance aware multi-label learning. IEEE Trans. Cybern. , 52(6):4459-4471, 2022.

Min-Ling Zhang and Lei Wu. Lift: Multi-label learning with label-specific features. IEEE Trans. Pattern Anal. Mach. Intell. , 37(1):107-120, 2015.

Min-Ling Zhang and Zhi-Hua Zhou. ML-KNN: A lazy learning approach to multi-label learning. Pattern Recognit. , 40(7):2038-2048, 2007.

Min-Ling Zhang and Zhi-Hua Zhou. A review on multi-label learning algorithms. IEEE transactions on knowledge and data engineering , 26(8):1819-1837, 2013.

Qian-Wen Zhang, Yun Zhong, and Min-Ling Zhang. Feature-induced labeling information enrichment for multi-label learning. In Proceedings of the 32th AAAI Conference on Artificial Intelligence , pages 4446-4453, New Orleans, LA, 2018.

Yue Zhu, Kai Ming Ting, and Zhi-Hua Zhou. Multi-label learning with emerging new labels. IEEE Trans. Knowl. Data Eng. , 30(10):1901-1914, 2018.

A Appendix

A.1 Experimental Results

Table 4, 5, 7 and 9 report detailed experimental results in terms of Average precision , Macroaveraging AUC , Coverage and One-error , which are not covered in the Experimental Results part of the main body due to page limit. Besides, the pairwise t-test is conducted to demonstrate whether the performance of SLAN is statistically superior/inferior to the comparing approaches on each data set. The resulting win/tie/loss counts in terms of multi-label evaluation metrics are summarized in Table 6 as well as F-measure in Table 8.

A.2 Data and Code Availability

Our source code can be found at https://palm.seu.edu.cn/zhangml/ . The datasets used in this paper are public, and can be found in Section 4.1.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We claim the paper's contributions and scope in the introduction (Section 1).

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discussed the limitations of this paper in the conclusion (Section 5).

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please refer to Section 4.1 and A.1 of this paper.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The paper provides the data and code availability in Section A.2.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please refer to Section 4.1 and A.1 of this paper.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: In Section 4.1 and A.1 of this paper, we presented the mean and standard deviation of multiple training rounds and conduct pairwise t-test between the proposed approach and the comparing approaches.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please refer to Section 4.1.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have already reviewed the NeurIPS Code of Ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work is foundational research and does not involve societal impact.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not pose such risk.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of classifying instances with unknown labels in a multi-label setting (multi-label open set recognition). A\nnovel approach named SLAN is proposed which leverages sub-labeling information, and unknown labels are recognized by differentiating the sub-labeling information from holistic supervision. The effectiveness of the proposed method is validated on different benchmarks.\n\nSTRENGTHS:\n- The paper presents a detailed and well-formulated methodology, including the method for structural information discovery and the part related to optimization.\n\n- Extensive experiments on various datasets are conducted to prove the effectiveness of the proposed SLAN approach. The empirical results demonstrate superior or comparable performance to the mentioned methods.\n\nWEAKNESSES:\n- The paper introduces the MLOSR problem, which is novel, but it may overlap with existing tasks like zero-shot multi-label classification and open-vocabulary multi-label classification.\n\n- This paper has limited baselines for comparison. While the paper includes comparisons with several multi-label learning and anomaly detection approaches, it should also compare with state-of-the-art methods specifically designed for zero-shot and open-vocabulary multi-label classification.\n\n- The proposed SLAN approach need to solve multiple optimization problems, which may not scale well to large datasets.\n\nQUESTIONS:\n- There are existing tasks with similar settings. For example, in image tasks, there are zero-shot multi-label classification, and open vocabulary multi-label classification. What's the difference between the proposed task and these tasks? Is the setting more applicable?\n\n- It would be better if the paper can include state-of-the-art methods for zero-shot and open-vocabulary multi-label classification in the experimental comparison.\n\n- It would be better to include a flowchart outlining the key step in the proposed approach, which can improve the clarity and make it easier to follow.\n\nLIMITATIONS:\nThe limitation and societal impact is addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article introduces a new problem in multi-label open set recognition (MLOSR) and proposes a novel approach named Sub-Labeling Information Reconstruction for MLOSR (SLAN). SLAN utilizes sub-labeling information enriched by structural details in the feature space. Experimental results across various datasets demonstrate its effectiveness in addressing this new challenge of MLOSR.\n\nSTRENGTHS:\n1. This article focuses on a novel problem within multi-label open set recognition (MLOSR), expanding beyond traditional multi-label learning by tackling the identification of unknown open labels. This problem is particularly relevant in practical applications due to the dynamic and open nature of real-world environments.\n2. The strength of the paper lies in its clear and thorough analysis and description of the SLAN method.\n3. The analysis of SLAN across various experimental metrics on different datasets is comprehensive, providing a robust demonstration of its effectiveness in the experiments.\n4. The paper's process and structure are concise, making it relatively easy to understand.\n\nWEAKNESSES:\n1. In the Parameter Sensitivity Analysis section, there is some explanation regarding the experimental parameter settings, but it primarily involves varying each parameter individually without further elaboration on their combined effects or interactions. It would be beneficial to include more extensive experimental analysis that considers the interactions between different parameter combinations.\n2. The paper asserts that OC-SVM, IFOREST, and MUENLFOREST do not exhibit performance on par with SLAN. It would be advantageous to provide further elaboration on why the parameters of these comparison methods, particularly MUENLFOREST, are configured as they are.  \n3. Is there any insight into how SLAN performs differently compared to other methods on different datasets? Could this difference be related to dataset characteristics and distributions?\n\nQUESTIONS:\n1. There is concern regarding the robustness of the method. How stable is its performance when errors or irrelevant labels appear in the dataset?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe abstract discusses multi-label learning where instances can have multiple labels simultaneously. Traditional approaches assume a closed set scenario where test data labels are predefined during training. However, in real-world situations, new labels can emerge during testing, creating an open and dynamic environment. The paper explores Multi-Label Open Set Recognition (MLOSR), which tackles the challenge of recognizing instances with unknown labels in multi-label settings. The proposed approach, SLAN, utilizes sub-labeling information enriched by structural features in the data space to predict unknown labels. Experimental results across different datasets demonstrate the effectiveness of SLAN in addressing the MLOSR problem.\n\nSTRENGTHS:\n1. The paper is well-written and easy to follow.  It shows comprehensive analyses of the motivation of the studied problem and the proposed method.\n2.  The paper introduces a novel learning framework called Multi-Label Open Set Recognition (MLOSR), which provides a new direction for multi-label learning research. To solve the MLOSR problem, an approach named SLAN that utilizes sub-labeling information and holistic supervision to recognize unknown labels is proposed. The techniques of SLAN are sound and well-motivated,  offering an effective solution for open set recognition in a multi-label environment. \n3. The paper conducts comprehensive empirical validations on various datasets, showing that the proposed method achieves superior or at least comparable performance across multiple evaluation metrics， which enhances the credibility of the research. The paper includes ah findings and indicates its robustness in multi-label learning.  Extensive sensitivity analysis of the trade-off parameters in the SLAN algorithm are conducted, guiding parameter selection in practical applications.\n\nWEAKNESSES:\nGeneralization Ability: The SLAN algorithm may have limited generalization ability when dealing with extremely multi-label datasets, restricting its application in broader scenarios.\nFeature Representation Limitations: Operating within the multi-label learning framework, SLAN might be constrained by less informative or discriminative feature representations.\nComputational Resources: While the paper mentions the computational resources for the experiments, it does not elaborate on the computational efficiency and scalability of the algorithm, which may impact its application to large-scale datasets.\nTheoretical Foundation: The paper does not provide theoretical results or proofs, which might reduce the depth of understanding of the algorithm's performance.\nAccess to Code and Datasets: Although the datasets are public, the paper states that the code will be released after acceptance, which may temporarily limit the reproducibility of the results.\n\nQUESTIONS:\nHow does the SLAN algorithm perform on datasets with highly imbalanced label distributions?\nDoes the parameter sensitivity analysis mentioned in the paper consider the characteristics of different types of datasets?\nHow adaptable and efficient is the SLAN algorithm for real-time or dynamically changing data streams in practical applications?\nDoes the paper consider the interpretability of the algorithm, i.e., how to explain the predictive results of SLAN?\nFor future work, are the authors planning to combine the SLAN algorithm with deep learning models to enhance the quality of feature representation and the performance of the algorithm?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}