# Multilabel Classification Using Low-Rank Decomposition

Authors: Bo Yang; Kunkun Tong; Xueqing Zhao; Shanmin Pang; Jinguang Chen
Journal: Discrete Dynamics in Nature and Society
 (2020)

---

## Abstract

In the multilabel learning framework, each instance is no longer associated with a single semantic, but rather with concept ambiguity. Specifically, the ambiguity of an instance in the input space means that there are multiple corresponding labels in the output space. In most of the existing multilabel classification methods, a binary annotation vector is used to denote the multiple semantic concepts. That is, +1 denotes that the instance has a relevant label, while −1 means the opposite. However, the label representation contains too little semantic information to truly express the differences among multiple different labels. Therefore, we propose a new approach to transform binary label into a real-valued label. We adopt the low-rank decomposition to get latent label information and then incorporate the information and original features to generate new features. Then, using the sparse representation to reconstruct the new instance, the reconstruction error can also be applied in the label space. In this way, we finally achieve the purpose of label conversion. Extensive experiments validate that the proposed method can achieve comparable to or even better results than other state-of-the-art algorithms.


---

## Body

## 1. Introduction

Classification is a high-frequency vocabulary in machine learning. We often say that classification generally refers to single-label classification, that is, an object is given a category. In multilabel learning, the meaning of classification is multilabel classification. Specifically, an instance is associated with more than one class label simultaneously. Multilabel learning has many application fields, such as web mining [1–3], text categorization [4–6], multimedia contents annotation [7–11], and bioinformatics [12–14].In recent years, the field of multilabel learning has gradually attracted significant attention. A variety of algorithms have been proposed, which can be basically divided into two categories [15]: algorithm adaptation and problem transformation. The core idea of the former is to transform the previous supervised learning algorithm so that it can be used to solve multilabel learning problems, such as ML-kNN [16], while the latter is to convert the multilabel learning problem into other known problems to solve, such as BR [17]. Some multilabel algorithms solve the multilabel learning problem without using the correlation among different labels, such as LIFT [18]. The main idea of the LIFT is to obtain the identifying characteristics of each label and build a new feature space. It first obtains the positive and negative examples corresponding to each label and then performs cluster analysis on the corresponding set of examples to obtain the cluster centers and finally uses the cluster centers to construct the label-specific features. In the process of solving the multilabel learning problem, LIFT does not consider label correlations; hence, it can be regarded as a new feature conversion method. Some algorithms consider the label correlation [19–25] for solving the multilabel learning problem. For example, the basic idea in [20] is to model the correlation among labels based on the Bayesian network and to achieve efficient learning by using the approximate strategy. Indeed, the rational use of the correlation among labels can effectively boost the performance of multilabel classification. For example, if an image has labels “football” and “rainforest,” it is likely to be labeled “Brazil”. It has a low probability of being labeled “river” if a document is annotated with “desert”. Therefore, how to effectively explore and make full use of label correlations is a crucial problem for multilabel learning.In fact, for an object with multiple labels, the importance of the related labels is still different. Although the importance of each label is not given directly, we can judge the importance of each label through external observation. Generally speaking, the larger the proportion in the original object, the more important the corresponding label. Accordingly, how to accurately express the importance of the label is also a challenge.The method in [26] decomposes the original output space in order to obtain potential label semantic information, which can effectively increase the ability of the subsequent feature selection. Motivated by the decomposition of the label space in [26], in the paper, we propose a method named label low-rank decomposition (LLRD) for multilabel classification. The LLRD algorithm first performs low-rank decomposition on the label matrix, then combines the decomposed results with the original features to form new features, and mines the structural information of the feature through sparse reconstruction. Third, it transforms the binary label into the real-valued and finally converts the classification problem into a regression problem.The contribution of this paper is as follows:(1)
Utilize low-rank decomposition to reveal the global label correlations and achieve good classification results(2)
Combine the low-rank decomposition results with the original features reducing the information loss in the subsequent label transformation process(3)
Carry out extensive experiments on different field datasets to verify the effectiveness of different algorithms


## 2. Materials and Methods

### 2.1. Datasets

In this experiment, a total of 13 datasets were used covering four fields: audio, text, image, and biology. All these data resources can be collected from Mulan (http://mulan.sourceforge.net/datasets.html) and Meka (http://meka.sourceforge.net/#datasetsru). Table 1 gives the specific details of the datasets. The number of instances, label space, and the dimension of features are denoted by |S|, L(S), and D(S), respectively. LDen (S) is the density of label, which is the result of the normalization of label cardinality LCard(S).Table 1
Properties of the experimental datasets.
Datasets
|S|
D(S)
L(S)
LCard(S)
LDen(S)
Domain
cal500
502
68
174
26.044
0.150
Audio
Emotions
593
72
6
1.868
0.311
Audio
Medical
978
1449
45
1.245
0.028
Text
Llog
1460
1004
75
1.180
0.016
Text
Image
2000
294
5
1.236
0.247
Image
Scene
2407
294
5
1.074
0.179
Image
Yeast
2417
103
14
4.237
0.303
Biology
Slashdot
3782
1079
22
1.181
0.054
Text
rcv1subset1
6000
500
101
2.880
0.029
Text
rcv1subset2
6000
500
101
2.634
0.026
Text
rcv1subset3
6000
500
101
2.614
0.026
Text
rcv1subset4
6000
500
101
2.484
0.025
Text
rcv1subset5
6000
500
101
2.642
0.026
Text


### 2.2. Notations

Formally, supposeX=ℝd be the d-dimensional input space and Y=l1,l2,…,lq denote the output domain of q class labels. Let D=xi,yi|1≤i≤p be the multilabel training dataset with p examples, where xi∈X is a d-dimensional instance vector and yi⊆Y is the label vector corresponding to xi. Let X=x1,x2,…,xp∈ℝd×p represent the input data matrix, and Xi=x1,x2,…,xi−1,xi+1,…,xp denote the matrix from which xi is removed from X. Let Y=y1,y2,⋯,yp∈−1,1q×p is a matrix composed of label vector.


### 2.3. The Process of LLRD

First, LLRD decomposes the label matrix with low-rank method. In the framework of multilabel learning, label matrix is often considered to be low rank [27, 28] due to the existence of label correlations. Low-rank structure is also a way to explore the global relationship between labels. Therefore, we can perform low-rank decomposition on the label matrix. Assuming that the rank of Y is r < q, Y can be written as follows:(1)Y≃AB,where A∈ℝq×r represents the dependency of B∈ℝr×p on the original label space and B is a mapping of the original label and also contains label correlation information.Second, we combineB with X to form a new feature space N=X;Bn1,n2,…,np=∈ℝr+d×p. In order to reveal the inner structure of the feature space, we use sparse reconstruction [29] method to model the relationship between the training instances. Specifically, we use Wsij=p×p to represent the training object relationship matrix, where sij is a measure of the relationship between ni and nj. Let Si=s1i,…,si−1,i,si+1,i,…,spiT denote the corresponding sparse reconstruction coefficient related to ni. According to the sparse representation theory, Si can be calculated as follows:(2)minSiNiSi−ni22+ηSi1,where Ni=n1,n2,…,ni−1,ni+1,…,np represent a combination of all training instances except ni. We can solve the above problem using alternating direction method of multiplier [30].Third, we transform the original binary label setyi=li1,li2,…,liqT associated with any xi in the training set into a real-valued label vector ci=ci1,ci2,…,ciqT, where lij∈−1,1 and cij∈ℝ. Because the real value contains more information, and through the size of the value, we can also infer the importance of the label. Since the input space and the label space are often interrelated, it is assumed that the relationship between ni and nj in the input space also exists between ci and cj in the label space. Accordingly, the representation errors of different elements in the label space can be written as follows:(3)minC∑i=1pci−∑j=1psijcj22s.t.k1≤lijcij≤k21≤i≤p,1≤j≤q,where c=c1,c2,…,cp. The above quadratic programming problem can be solved by mature tools related to quadratic programming. The original multilabel classification problem can be transferred into a multioutput regression problem. There are many solutions [31] to solve it. The learning of LLRD method contains three phases: low-rank decomposition, sparse reconstruction, and multioutput regression. The time complexity of low-rank decomposition and sparse reconstruction is Od2p+d3. If we choose multioutput support vector regression to realize the classification, the time complexity is Oqp3. Thus, the total complexity of LLRD is Od2p+d3+qp3.


## 2.1. Datasets

In this experiment, a total of 13 datasets were used covering four fields: audio, text, image, and biology. All these data resources can be collected from Mulan (http://mulan.sourceforge.net/datasets.html) and Meka (http://meka.sourceforge.net/#datasetsru). Table 1 gives the specific details of the datasets. The number of instances, label space, and the dimension of features are denoted by |S|, L(S), and D(S), respectively. LDen (S) is the density of label, which is the result of the normalization of label cardinality LCard(S).Table 1
Properties of the experimental datasets.
Datasets
|S|
D(S)
L(S)
LCard(S)
LDen(S)
Domain
cal500
502
68
174
26.044
0.150
Audio
Emotions
593
72
6
1.868
0.311
Audio
Medical
978
1449
45
1.245
0.028
Text
Llog
1460
1004
75
1.180
0.016
Text
Image
2000
294
5
1.236
0.247
Image
Scene
2407
294
5
1.074
0.179
Image
Yeast
2417
103
14
4.237
0.303
Biology
Slashdot
3782
1079
22
1.181
0.054
Text
rcv1subset1
6000
500
101
2.880
0.029
Text
rcv1subset2
6000
500
101
2.634
0.026
Text
rcv1subset3
6000
500
101
2.614
0.026
Text
rcv1subset4
6000
500
101
2.484
0.025
Text
rcv1subset5
6000
500
101
2.642
0.026
Text


## 2.2. Notations

Formally, supposeX=ℝd be the d-dimensional input space and Y=l1,l2,…,lq denote the output domain of q class labels. Let D=xi,yi|1≤i≤p be the multilabel training dataset with p examples, where xi∈X is a d-dimensional instance vector and yi⊆Y is the label vector corresponding to xi. Let X=x1,x2,…,xp∈ℝd×p represent the input data matrix, and Xi=x1,x2,…,xi−1,xi+1,…,xp denote the matrix from which xi is removed from X. Let Y=y1,y2,⋯,yp∈−1,1q×p is a matrix composed of label vector.


## 2.3. The Process of LLRD

First, LLRD decomposes the label matrix with low-rank method. In the framework of multilabel learning, label matrix is often considered to be low rank [27, 28] due to the existence of label correlations. Low-rank structure is also a way to explore the global relationship between labels. Therefore, we can perform low-rank decomposition on the label matrix. Assuming that the rank of Y is r < q, Y can be written as follows:(1)Y≃AB,where A∈ℝq×r represents the dependency of B∈ℝr×p on the original label space and B is a mapping of the original label and also contains label correlation information.Second, we combineB with X to form a new feature space N=X;Bn1,n2,…,np=∈ℝr+d×p. In order to reveal the inner structure of the feature space, we use sparse reconstruction [29] method to model the relationship between the training instances. Specifically, we use Wsij=p×p to represent the training object relationship matrix, where sij is a measure of the relationship between ni and nj. Let Si=s1i,…,si−1,i,si+1,i,…,spiT denote the corresponding sparse reconstruction coefficient related to ni. According to the sparse representation theory, Si can be calculated as follows:(2)minSiNiSi−ni22+ηSi1,where Ni=n1,n2,…,ni−1,ni+1,…,np represent a combination of all training instances except ni. We can solve the above problem using alternating direction method of multiplier [30].Third, we transform the original binary label setyi=li1,li2,…,liqT associated with any xi in the training set into a real-valued label vector ci=ci1,ci2,…,ciqT, where lij∈−1,1 and cij∈ℝ. Because the real value contains more information, and through the size of the value, we can also infer the importance of the label. Since the input space and the label space are often interrelated, it is assumed that the relationship between ni and nj in the input space also exists between ci and cj in the label space. Accordingly, the representation errors of different elements in the label space can be written as follows:(3)minC∑i=1pci−∑j=1psijcj22s.t.k1≤lijcij≤k21≤i≤p,1≤j≤q,where c=c1,c2,…,cp. The above quadratic programming problem can be solved by mature tools related to quadratic programming. The original multilabel classification problem can be transferred into a multioutput regression problem. There are many solutions [31] to solve it. The learning of LLRD method contains three phases: low-rank decomposition, sparse reconstruction, and multioutput regression. The time complexity of low-rank decomposition and sparse reconstruction is Od2p+d3. If we choose multioutput support vector regression to realize the classification, the time complexity is Oqp3. Thus, the total complexity of LLRD is Od2p+d3+qp3.


## 3. Results and Discussion

### 3.1. Experiment Setup

In this subsection, we investigate comparisons between our LLRD and other six multilabel learning methods on six multilabel evaluation criteria, which include two categories: example-based and label-based metrics [32]. The example-based metric is to first obtain the performance of the learning system on each test example and finally returns the average of the entire test set. Unlike the above example-based metric, the label-based metric first returns the performance of the system on each label and finally gets the macro/microaveraged F1 value on all labels.In this paper,one-error, coverage, ranking loss, and average precision are employed for example-based performance evaluation. And macroaveraging and microaveraging F1 are label-based metrics. For example-based metrics except average precision, as their values increase, it means that the performance of the algorithm is worse. For the remaining metrics, their values are proportional to the performance of the algorithm.LetT=xi,yii=1m⊂ℝd×+1,−1q be the multilabel test set and fx,l can be seen as the confidence of l being the corresponding label associating with x. In addition, fx,l can be converted into a ranking function rankfx,l. If fx,l1>fx,l2 holds, then the corresponding ranking function has rankfx,l1<rankfx,l2.The six evaluation criteria for the algorithm used in the paper are defined as follows:(1)
One-error:(4)one‐errorf=1m∑i=1margmaxl∈Yfxi,l∉yi.(2)
Coverage:(5)coveragef=1m∑i=1mmaxl∈yirankfxi,l−1.(3)
Ranking loss:(6)rlossf=1m∑i=1m1yiy¯il′,l′′|fxi,l′≤fxi,l′′,l′,l′′∈yi×y¯i.(4)
Average precision:(7)avgprecf=1m∑i=1m1yi∑l∈yil′|rankfx,l′≤rankfxi,l,l′∈yirankfxi,l.(5)
Macroaveraging F1:(8)F1macroh=1q∑j=1q2TPj2TPj+FNj+FPj.(6)
Microaveraging F1:(9)F1microh=2∑i=1qTPj2∑i=1qTPj+∑i=1qFNj+∑i=1qFPj,where FNj, TNj, FPj, and TPj indicate the number of false-negative, true-negative, false-positive, and true-positive instances with regard to lj.In order to test the effectiveness of LLRD, we chose six multilabel learning algorithms MLFE [33], RAKEL [34], ML2 [35], CLR [36], LIFT [18], and RELIAB [37] for performance comparison. MLFE makes full use of the intrinsic information in feature space, making the semantics of the label space more abundant. The specific parameters of MLFE are set as follows: ρ=1, c1=1, c2=2, and β1, β2, and β3 searched from {1, 2, …,10}, {1, 10, 15}, and {1, 10}. RAKEL is a high-order approach. The basic idea of the algorithm is to transform the multilabel learning problem into integration of multiclass classification problem. We use the default settings recommended by RAKEL algorithm, namely, k=3, ensemble size n=2q. For ML2, respective parameter values are recorded as follows: λ=1,K=l+1 and C1 and C2 selected from {1, 2, …, 10}. ML2 is the first multilabel learning algorithm to attempt to explore manifolds at the label level. CLR is a second-order problem transformation method. It solves the problem of multilabel classification by using label ranking, in which ranking among labels is implemented by pairwise comparison. The associated parameter ensemble size is set to 2q. LIFT uses different feature sets to distinguish different labels by clustering positive and negative examples. The value of ratio parameter r is 0.1, as suggested in [18]. RELIAB utilizes the implicit relative information of label to achieve the task of multilabel learning. The parameters τ and β take values from {0.1, 0.15, …, 0.5} and {0.001, 0.01, …, 10}, respectively. For LLRD, η=1, r can be selected from {1, 2, …, q−1}. In a word, the parameter settings of the comparison algorithm are as recommended in the related papers.


### 3.2. Experimental Results

For each dataset in our experiment, we adopt the tenfold cross-validation strategy. Our experimental results are mainly distributed in Tables2 and 3, where we record the performance of different algorithms in different multilabel datasets. Specifically, the average and standard deviation of the corresponding evaluation criteria are recorded in the tables. For each evaluation metric, “↓” indicates “the smaller the better” and “↑” indicates “the larger the better”. The best results are shown in bold form.Table 2
Performance of each multilabel algorithm (mean ± std. deviation) on the regular-scale datasets.
Comparing algorithms
cal500
Emotions
Medical
Llog
Image
Scene
Yeast
One-error ↓
LLRD
0.136 ± 0.041
0.248 ± 0.048
0.125 ± 0.031
0.657 ± 0.038
0.244 ± 0.018
0.116 ± 0.019
0.217 ± 0.013
MLFE
0.168 ± 0.049
0.259 ± 0.050
0.131 ± 0.030
0.672 ± 0.041
0.257 ± 0.031
0.127 ± 0.022
0.233 ± 0.026
LIFT
0.125 ± 0.049
0.251 ± 0.027
0.156 ± 0.041
0.664 ± 0.034
0.276 ± 0.026
0.132 ± 0.012
0.226 ± 0.021
RELIAB
0.116 ± 0.030
0.255 ± 0.041
0.163 ± 0.028
0.754 ± 0.035
0.342 ± 0.032
0.258 ± 0.011
0.255 ± 0.016
ML2
0.201 ± 0.090
0.261 ± 0.045
0.135 ± 0.032
0.674 ± 0.051
0.260 ± 0.027
0.144 ± 0.019
0.246 ± 0.034
CLR
0.243 ± 0.058
0.310 ± 0.019
0.362 ± 0.009
0.841 ± 0.036
0.449 ± 0.013
0.331 ± 0.031
0.234 ± 0.022
RAKEL
0.622 ± 0.065
0.289 ± 0.032
0.237 ± 0.032
0.871 ± 0.028
0.397 ± 0.019
0.314 ± 0.030
0.291 ± 0.031
Coverage ↓
LLRD
0.774 ± 0.021
0.282 ± 0.034
0.029 ± 0.009
0.194 ± 0.025
0.157 ± 0.010
0.008 ± 0.009
0.447 ± 0.010
MLFE
0.769 ± 0.024
0.283 ± 0.030
0.033 ± 0.010
0.200 ± 0.027
0.162 ± 0.018
0.012 ± 0.008
0.449 ± 0.011
LIFT
0.753 ± 0.015
0.271 ± 0.023
0.040 ± 0.014
0.164 ± 0.007
0.172 ± 0.013
0.026 ± 0.007
0.454 ± 0.017
RELIAB
0.746 ± 0.019
0.306 ± 0.020
0.044 ± 0.013
0.155 ± 0.013
0.185 ± 0.007
0.114 ± 0.004
0.457 ± 0.015
ML2
0.814 ± 0.033
0.292 ± 0.044
0.035 ± 0.013
0.201 ± 0.026
0.164 ± 0.009
0.010 ± 0.007
0.461 ± 0.016
CLR
0.789 ± 0.010
0.330 ± 0.011
0.073 ± 0.041
0.182 ± 0.050
0.233 ± 0.017
0.122 ± 0.011
0.484 ± 0.020
RAKEL
0.958 ± 0.011
0.335 ± 0.031
0.077 ± 0.014
0.332 ± 0.021
0.249 ± 0.006
0.161 ± 0.007
0.553 ± 0.016
Ranking loss ↓
LLRD
0.185 ± 0.011
0.144 ± 0.028
0.018 ± 0.007
0.185 ± 0.022
0.129 ± 0.010
0.042 ± 0.008
0.163 ± 0.008
MLFE
0.188 ± 0.010
0.146 ± 0.030
0.014 ± 0.007
0.191 ± 0.025
0.134 ± 0.017
0.046 ± 0.010
0.167 ± 0.011
LIFT
0.178 ± 0.008
0.144 ± 0.026
0.029 ± 0.009
0.148 ± 0.014
0.148 ± 0.012
0.054 ± 0.015
0.164 ± 0.013
RELIAB
0.182 ± 0.007
0.165 ± 0.021
0.026 ± 0.008
0.134 ± 0.011
0.176 ± 0.008
0.076 ± 0.007
0.185 ± 0.021
ML2
0.205 ± 0.021
0.153 ± 0.033
0.011 ± 0.009
0.194 ± 0.027
0.136 ± 0.012
0.050 ± 0.007
0.175 ± 0.015
CLR
0.231 ± 0.020
0.181 ± 0.020
0.072 ± 0.051
0.137 ± 0.028
0.241 ± 0.015
0.098 ± 0.021
0.196 ± 0.009
RAKEL
0.359 ± 0.012
0.213 ± 0.019
0.066 ± 0.019
0.281 ± 0.034
0.244 ± 0.016
0.155 ± 0.023
0.243 ± 0.010
Average precision ↑
LLRD
0.506 ± 0.018
0.819 ± 0.031
0.905 ± 0.020
0.421 ± 0.033
0.841 ± 0.009
0.934 ± 0.010
0.775 ± 0.008
MLFE
0.490 ± 0.017
0.812 ± 0.032
0.901 ± 0.021
0.410 ± 0.029
0.835 ± 0.019
0.928 ± 0.013
0.766 ± 0.016
LIFT
0.502 ± 0.021
0.824 ± 0.024
0.880 ± 0.030
0.416 ± 0.031
0.820 ± 0.018
0.922 ± 0.008
0.768 ± 0.018
RELIAB
0.497 ± 0.016
0.801 ± 0.021
0.869 ± 0.020
0.405 ± 0.041
0.781 ± 0.009
0.851 ± 0.008
0.751 ± 0.010
ML2
0.481 ± 0.030
0.816 ± 0.031
0.898 ± 0.022
0.404 ± 0.031
0.832 ± 0.014
0.930 ± 0.009
0.759 ± 0.020
CLR
0.425 ± 0.034
0.770 ± 0.019
0.695 ± 0.032
0.312 ± 0.059
0.722 ± 0.015
0.801 ± 0.012
0.755 ± 0.006
RAKEL
0.343 ± 0.009
0.772 ± 0.037
0.798 ± 0.018
0.228 ± 0.020
0.731 ± 0.017
0.777 ± 0.023
0.717 ± 0.007
MacroaveragingF1 ↑
LLRD
0.231 ± 0.026
0.676 ± 0.051
0.736 ± 0.050
0.408 ± 0.028
0.666 ± 0.024
0.800 ± 0.016
0.420 ± 0.030
MLFE
0.239 ± 0.025
0.668 ± 0.050
0.702 ± 0.056
0.415 ± 0.041
0.655 ± 0.021
0.787 ± 0.015
0.430 ± 0.024
LIFT
0.179 ± 0.014
0.651 ± 0.035
0.694 ± 0.052
0.392 ± 0.045
0.624 ± 0.033
0.788 ± 0.018
0.377 ± 0.019
RELIAB
0.288 ± 0.015
0.639 ± 0.038
0.686 ± 0.058
0.394 ± 0.031
0.568 ± 0.030
0.671 ± 0.021
0.409 ± 0.023
ML2
0.226 ± 0.024
0.656 ± 0.045
0.686 ± 0.058
0.382 ± 0.035
0.652 ± 0.018
0.783 ± 0.015
0.438 ± 0.017
CLR
0.220 ± 0.017
0.604 ± 0.032
0.616 ± 0.118
0.402 ± 0.056
0.523 ± 0.027
0.635 ± 0.013
0.386 ± 0.016
RAKEL
0.195 ± 0.010
0.615 ± 0.030
0.679 ± 0.037
0.377 ± 0.054
0.545 ± 0.018
0.654 ± 0.012
0.441 ± 0.011
MicroaveragingF1 ↑
LLRD
0.325 ± 0.011
0.692 ± 0.048
0.814 ± 0.030
0.126 ± 0.027
0.665 ± 0.024
0.792 ± 0.017
0.656 ± 0.011
MLFE
0.384 ± 0.017
0.683 ± 0.047
0.785 ± 0.031
0.137 ± 0.032
0.653 ± 0.024
0.781 ± 0.015
0.643 ± 0.013
LIFT
0.313 ± 0.013
0.664 ± 0.015
0.763 ± 0.031
0.168 ± 0.034
0.625 ± 0.031
0.779 ± 0.022
0.650 ± 0.016
RELIAB
0.454 ± 0.011
0.647 ± 0.038
0.748 ± 0.024
0.188 ± 0.028
0.562 ± 0.021
0.639 ± 0.013
0.631 ± 0.015
ML2
0.366 ± 0.013
0.674 ± 0.042
0.780 ± 0.021
0.074 ± 0.031
0.650 ± 0.019
0.776 ± 0.018
0.635 ± 0.018
CLR
0.330 ± 0.012
0.626 ± 0.029
0.606 ± 0.143
0.165 ± 0.050
0.531 ± 0.008
0.634 ± 0.017
0.623 ± 0.010
RAKEL
0.356 ± 0.025
0.648 ± 0.024
0.669 ± 0.016
0.155 ± 0.019
0.533 ± 0.005
0.645 ± 0.009
0.637 ± 0.011Table 3
Performance of each multilabel algorithm (mean ± std. deviation) on the large-scale datasets.
Comparing algorithms
Slashdot
rcv1subset1
rcv1subset2
rcv1subset3
rcv1subset4
rcv1subset5
One-error ↓
LLRD
0.363 ± 0.026
0.414 ± 0.013
0.411 ± 0.017
0.416 ± 0.029
0.317 ± 0.015
0.401 ± 0.018
MLFE
0.374 ± 0.027
0.406 ± 0.018
0.399 ± 0.013
0.402 ± 0.025
0.328 ± 0.013
0.392 ± 0.008
LIFT
0.393 ± 0.033
0.427 ± 0.011
0.434 ± 0.017
0.441 ± 0.020
0.363 ± 0.019
0.430 ± 0.019
RELIAB
0.508 ± 0.022
0.449 ± 0.015
0.458 ± 0.028
0.454 ± 0.012
0.433 ± 0.024
0.423 ± 0.009
ML2
0.370 ± 0.025
0.404 ± 0.017
0.395 ± 0.018
0.398 ± 0.021
0.323 ± 0.021
0.388 ± 0.010
CLR
0.965 ± 0.013
0.513 ± 0.022
0.515 ± 0.009
0.518 ± 0.028
0.472 ± 0.031
0.521 ± 0.021
RAKEL
0.602 ± 0.009
0.605 ± 0.013
0.574 ± 0.012
0.585 ± 0.022
0.561 ± 0.022
0.614 ± 0.009
Coverage ↓
LLRD
0.107 ± 0.010
0.125 ± 0.008
0.121 ± 0.009
0.123 ± 0.006
0.092 ± 0.004
0.116 ± 0.009
MLFE
0.126 ± 0.013
0.136 ± 0.005
0.130 ± 0.010
0.129 ± 0.007
0.094 ± 0.007
0.124 ± 0.007
LIFT
0.112 ± 0.008
0.144 ± 0.020
0.135 ± 0.008
0.156 ± 0.008
0.113 ± 0.012
0.148 ± 0.013
RELIAB
0.131 ± 0.007
0.152 ± 0.012
0.128 ± 0.014
0.144 ± 0.011
0.105 ± 0.020
0.131 ± 0.014
ML2
0.103 ± 0.011
0.138 ± 0.008
0.132 ± 0.010
0.126 ± 0.006
0.078 ± 0.006
0.129 ± 0.009
CLR
0.254 ± 0.003
0.146 ± 0.018
0.141 ± 0.007
0.137 ± 0.010
0.109 ± 0.018
0.136 ± 0.011
RAKEL
0.226 ± 0.020
0.426 ± 0.023
0.372 ± 0.016
0.381 ± 0.014
0.365 ± 0.009
0.388 ± 0.020
Ranking loss ↓
LLRD
0.090 ± 0.010
0.049 ± 0.004
0.050 ± 0.004
0.052 ± 0.002
0.038 ± 0.002
0.047 ± 0.003
MLFE
0.107 ± 0.013
0.052 ± 0.002
0.055 ± 0.007
0.055 ± 0.002
0.040 ± 0.004
0.050 ± 0.003
LIFT
0.098 ± 0.016
0.058 ± 0.007
0.057 ± 0.009
0.068 ± 0.004
0.059 ± 0.010
0.055 ± 0.007
RELIAB
0.124 ± 0.003
0.066 ± 0.010
0.063 ± 0.008
0.062 ± 0.004
0.052 ± 0.006
0.063 ± 0.005
ML2
0.103 ± 0.012
0.056 ± 0.004
0.057 ± 0.004
0.056 ± 0.003
0.031 ± 0.003
0.050 ± 0.004
CLR
0.237 ± 0.008
0.062 ± 0.011
0.066 ± 0.008
0.065 ± 0.012
0.047 ± 0.006
0.071 ± 0.005
RAKEL
0.211 ± 0.019
0.226 ± 0.019
0.215 ± 0.017
0.230 ± 0.015
0.235 ± 0.014
0.214 ± 0.016
Average precision ↑
LLRD
0.725 ± 0.019
0.611 ± 0.010
0.638 ± 0.011
0.634 ± 0.017
0.717 ± 0.008
0.643 ± 0.011
MLFE
0.712 ± 0.021
0.618 ± 0.016
0.645 ± 0.009
0.639 ± 0.014
0.708 ± 0.012
0.647 ± 0.012
LIFT
0.703 ± 0.010
0.586 ± 0.009
0.598 ± 0.012
0.595 ± 0.011
0.674 ± 0.013
0.598 ± 0.011
RELIAB
0.624 ± 0.014
0.578 ± 0.021
0.611 ± 0.011
0.614 ± 0.018
0.655 ± 0.018
0.604 ± 0.009
ML2
0.715 ± 0.022
0.621 ± 0.012
0.647 ± 0.013
0.643 ± 0.016
0.717 ± 0.013
0.650 ± 0.010
CLR
0.269 ± 0.002
0.575 ± 0.013
0.584 ± 0.021
0.571 ± 0.032
0.614 ± 0.020
0.588 ± 0.013
RAKEL
0.522 ± 0.020
0.395 ± 0.012
0.445 ± 0.018
0.431 ± 0.014
0.450 ± 0.012
0.437 ± 0.016
MacroaveragingF1 ↑
LLRD
0.427 ± 0.035
0.235 ± 0.020
0.259 ± 0.019
0.213 ± 0.031
0.300 ± 0.019
0.211 ± 0.020
MLFE
0.466 ± 0.035
0.198 ± 0.017
0.195 ± 0.056
0.202 ± 0.030
0.249 ± 0.021
0.204 ± 0.021
LIFT
0.429 ± 0.037
0.223 ± 0.025
0.186 ± 0.024
0.200 ± 0.031
0.238 ± 0.013
0.196 ± 0.031
RELIAB
0.425 ± 0.029
0.342 ± 0.022
0.338 ± 0.016
0.348 ± 0.014
0.342 ± 0.028
0.352 ± 0.014
ML2
0.472 ± 0.029
0.216 ± 0.020
0.206 ± 0.024
0.195 ± 0.030
0.244 ± 0.023
0.208 ± 0.011
CLR
0.174 ± 0.032
0.285 ± 0.032
0.264 ± 0.021
0.272 ± 0.022
0.311 ± 0.031
0.305 ± 0.017
RAKEL
0.354 ± 0.037
0.269 ± 0.030
0.251 ± 0.014
0.255 ± 0.014
0.263 ± 0.014
0.274 ± 0.018
MicroaveragingF1 ↑
LLRD
0.496 ± 0.021
0.393 ± 0.013
0.381 ± 0.017
0.406 ± 0.027
0.470 ± 0.013
0.402 ± 0.018
MLFE
0.545 ± 0.019
0.373 ± 0.014
0.375 ± 0.031
0.392 ± 0.024
0.403 ± 0.020
0.381 ± 0.017
LIFT
0.510 ± 0.030
0.320 ± 0.017
0.353 ± 0.014
0.347 ± 0.018
0.342 ± 0.024
0.363 ± 0.008
RELIAB
0.453 ± 0.011
0.408 ± 0.010
0.449 ± 0.008
0.451 ± 0.021
0.478 ± 0.016
0.454 ± 0.012
ML2
0.556 ± 0.022
0.371 ± 0.014
0.391 ± 0.010
0.383 ± 0.026
0.393 ± 0.022
0.410 ± 0.015
CLR
0.104 ± 0.005
0.367 ± 0.011
0.368 ± 0.024
0.320 ± 0.024
0.381 ± 0.015
0.372 ± 0.008
RAKEL
0.365 ± 0.020
0.359 ± 0.023
0.348 ± 0.016
0.341 ± 0.016
0.371 ± 0.015
0.342 ± 0.006We useFriedman test [38] based on the average ranks for verifying whether the difference between algorithms is statistically significant. If the assumption that “all algorithms have equal performance” is rejected, it means that the performance of each algorithm is significantly different. As can be seen from the data presented in Table 4, the hypothesis that there is no significant difference among the algorithms is not valid under the condition of 0.05 significance level. Therefore, we need to conduct a post hoc test to further distinguish the various algorithms. Usually, there are two options for post hoc test, one is the Nemenyi test [38] and the other is the Bonferroni–Dunn test [39]. For k algorithms, the former needs to compare kk−1/2 times, while the latter only needs k−1 times in some cases. Thus, we choose the latter. The Bonferroni–Dunn test is used to test whether LLRD is more competitive than the comparative algorithm, in which LLRD plays a role of control algorithm. When the difference of average rank between two algorithms is more than one critical difference CD, the performance of two algorithms is obviously different. The CD value mentioned here can be calculated from CD=qαkk+1/6N, where k = 7 and N = 13, when the significance level is 0.05, the corresponding qα=2.638.Table 4
The Friedman statisticsFF and the critical value.
Evaluation metric
F
F
Critical value
One-error
34.0909
2.2274
Coverage
20.3765
Ranking loss
21.1642
Average precision
39.8409
Macroaveraging F1
2.6520
Microaveraging F1
7.6088The CD diagram associated with LLRD and its comparison algorithm is shown in Figure1. The numbers on the horizontal axis of the coordinate indicate the average rank value of each algorithm under different evaluation criteria. There is no significant difference in performance among the various algorithms connected by solid lines.Figure 1
Comparison of LLRD (control algorithm) against other related approaches with the Bonferroni–Dunn test. Approaches that are not connected to LLRD are significantly different in performance from LLRD. (a) One-error. (b) Coverage. (c) Ranking loss. (d) Average precision. (e) MacroaveragingF1. (f) Microaveraging F2.
(a)
(b)
(c)
(d)
(e)
(f)Through the analysis of the above experimental results, we can draw the following conclusions:(1)
In terms of the four evaluation criteria ofone-error, coverage, ranking loss, and average precision, LLRD is obviously superior to RELIAB, RAKEL, and CLR.(2)
The smaller the average rank value, the better the performance of the corresponding. For LLRD, five of the average rank value in the six CD subdiagrams are optimal, which shows LLRD outperforms other algorithms.(3)
For regular-size datasets, LLRD ranks first in 69% of the cases under different evaluation criteria, while for large-scale datasets, it ranks first in 36.1%.


## 3.1. Experiment Setup

In this subsection, we investigate comparisons between our LLRD and other six multilabel learning methods on six multilabel evaluation criteria, which include two categories: example-based and label-based metrics [32]. The example-based metric is to first obtain the performance of the learning system on each test example and finally returns the average of the entire test set. Unlike the above example-based metric, the label-based metric first returns the performance of the system on each label and finally gets the macro/microaveraged F1 value on all labels.In this paper,one-error, coverage, ranking loss, and average precision are employed for example-based performance evaluation. And macroaveraging and microaveraging F1 are label-based metrics. For example-based metrics except average precision, as their values increase, it means that the performance of the algorithm is worse. For the remaining metrics, their values are proportional to the performance of the algorithm.LetT=xi,yii=1m⊂ℝd×+1,−1q be the multilabel test set and fx,l can be seen as the confidence of l being the corresponding label associating with x. In addition, fx,l can be converted into a ranking function rankfx,l. If fx,l1>fx,l2 holds, then the corresponding ranking function has rankfx,l1<rankfx,l2.The six evaluation criteria for the algorithm used in the paper are defined as follows:(1)
One-error:(4)one‐errorf=1m∑i=1margmaxl∈Yfxi,l∉yi.(2)
Coverage:(5)coveragef=1m∑i=1mmaxl∈yirankfxi,l−1.(3)
Ranking loss:(6)rlossf=1m∑i=1m1yiy¯il′,l′′|fxi,l′≤fxi,l′′,l′,l′′∈yi×y¯i.(4)
Average precision:(7)avgprecf=1m∑i=1m1yi∑l∈yil′|rankfx,l′≤rankfxi,l,l′∈yirankfxi,l.(5)
Macroaveraging F1:(8)F1macroh=1q∑j=1q2TPj2TPj+FNj+FPj.(6)
Microaveraging F1:(9)F1microh=2∑i=1qTPj2∑i=1qTPj+∑i=1qFNj+∑i=1qFPj,where FNj, TNj, FPj, and TPj indicate the number of false-negative, true-negative, false-positive, and true-positive instances with regard to lj.In order to test the effectiveness of LLRD, we chose six multilabel learning algorithms MLFE [33], RAKEL [34], ML2 [35], CLR [36], LIFT [18], and RELIAB [37] for performance comparison. MLFE makes full use of the intrinsic information in feature space, making the semantics of the label space more abundant. The specific parameters of MLFE are set as follows: ρ=1, c1=1, c2=2, and β1, β2, and β3 searched from {1, 2, …,10}, {1, 10, 15}, and {1, 10}. RAKEL is a high-order approach. The basic idea of the algorithm is to transform the multilabel learning problem into integration of multiclass classification problem. We use the default settings recommended by RAKEL algorithm, namely, k=3, ensemble size n=2q. For ML2, respective parameter values are recorded as follows: λ=1,K=l+1 and C1 and C2 selected from {1, 2, …, 10}. ML2 is the first multilabel learning algorithm to attempt to explore manifolds at the label level. CLR is a second-order problem transformation method. It solves the problem of multilabel classification by using label ranking, in which ranking among labels is implemented by pairwise comparison. The associated parameter ensemble size is set to 2q. LIFT uses different feature sets to distinguish different labels by clustering positive and negative examples. The value of ratio parameter r is 0.1, as suggested in [18]. RELIAB utilizes the implicit relative information of label to achieve the task of multilabel learning. The parameters τ and β take values from {0.1, 0.15, …, 0.5} and {0.001, 0.01, …, 10}, respectively. For LLRD, η=1, r can be selected from {1, 2, …, q−1}. In a word, the parameter settings of the comparison algorithm are as recommended in the related papers.


## 3.2. Experimental Results

For each dataset in our experiment, we adopt the tenfold cross-validation strategy. Our experimental results are mainly distributed in Tables2 and 3, where we record the performance of different algorithms in different multilabel datasets. Specifically, the average and standard deviation of the corresponding evaluation criteria are recorded in the tables. For each evaluation metric, “↓” indicates “the smaller the better” and “↑” indicates “the larger the better”. The best results are shown in bold form.Table 2
Performance of each multilabel algorithm (mean ± std. deviation) on the regular-scale datasets.
Comparing algorithms
cal500
Emotions
Medical
Llog
Image
Scene
Yeast
One-error ↓
LLRD
0.136 ± 0.041
0.248 ± 0.048
0.125 ± 0.031
0.657 ± 0.038
0.244 ± 0.018
0.116 ± 0.019
0.217 ± 0.013
MLFE
0.168 ± 0.049
0.259 ± 0.050
0.131 ± 0.030
0.672 ± 0.041
0.257 ± 0.031
0.127 ± 0.022
0.233 ± 0.026
LIFT
0.125 ± 0.049
0.251 ± 0.027
0.156 ± 0.041
0.664 ± 0.034
0.276 ± 0.026
0.132 ± 0.012
0.226 ± 0.021
RELIAB
0.116 ± 0.030
0.255 ± 0.041
0.163 ± 0.028
0.754 ± 0.035
0.342 ± 0.032
0.258 ± 0.011
0.255 ± 0.016
ML2
0.201 ± 0.090
0.261 ± 0.045
0.135 ± 0.032
0.674 ± 0.051
0.260 ± 0.027
0.144 ± 0.019
0.246 ± 0.034
CLR
0.243 ± 0.058
0.310 ± 0.019
0.362 ± 0.009
0.841 ± 0.036
0.449 ± 0.013
0.331 ± 0.031
0.234 ± 0.022
RAKEL
0.622 ± 0.065
0.289 ± 0.032
0.237 ± 0.032
0.871 ± 0.028
0.397 ± 0.019
0.314 ± 0.030
0.291 ± 0.031
Coverage ↓
LLRD
0.774 ± 0.021
0.282 ± 0.034
0.029 ± 0.009
0.194 ± 0.025
0.157 ± 0.010
0.008 ± 0.009
0.447 ± 0.010
MLFE
0.769 ± 0.024
0.283 ± 0.030
0.033 ± 0.010
0.200 ± 0.027
0.162 ± 0.018
0.012 ± 0.008
0.449 ± 0.011
LIFT
0.753 ± 0.015
0.271 ± 0.023
0.040 ± 0.014
0.164 ± 0.007
0.172 ± 0.013
0.026 ± 0.007
0.454 ± 0.017
RELIAB
0.746 ± 0.019
0.306 ± 0.020
0.044 ± 0.013
0.155 ± 0.013
0.185 ± 0.007
0.114 ± 0.004
0.457 ± 0.015
ML2
0.814 ± 0.033
0.292 ± 0.044
0.035 ± 0.013
0.201 ± 0.026
0.164 ± 0.009
0.010 ± 0.007
0.461 ± 0.016
CLR
0.789 ± 0.010
0.330 ± 0.011
0.073 ± 0.041
0.182 ± 0.050
0.233 ± 0.017
0.122 ± 0.011
0.484 ± 0.020
RAKEL
0.958 ± 0.011
0.335 ± 0.031
0.077 ± 0.014
0.332 ± 0.021
0.249 ± 0.006
0.161 ± 0.007
0.553 ± 0.016
Ranking loss ↓
LLRD
0.185 ± 0.011
0.144 ± 0.028
0.018 ± 0.007
0.185 ± 0.022
0.129 ± 0.010
0.042 ± 0.008
0.163 ± 0.008
MLFE
0.188 ± 0.010
0.146 ± 0.030
0.014 ± 0.007
0.191 ± 0.025
0.134 ± 0.017
0.046 ± 0.010
0.167 ± 0.011
LIFT
0.178 ± 0.008
0.144 ± 0.026
0.029 ± 0.009
0.148 ± 0.014
0.148 ± 0.012
0.054 ± 0.015
0.164 ± 0.013
RELIAB
0.182 ± 0.007
0.165 ± 0.021
0.026 ± 0.008
0.134 ± 0.011
0.176 ± 0.008
0.076 ± 0.007
0.185 ± 0.021
ML2
0.205 ± 0.021
0.153 ± 0.033
0.011 ± 0.009
0.194 ± 0.027
0.136 ± 0.012
0.050 ± 0.007
0.175 ± 0.015
CLR
0.231 ± 0.020
0.181 ± 0.020
0.072 ± 0.051
0.137 ± 0.028
0.241 ± 0.015
0.098 ± 0.021
0.196 ± 0.009
RAKEL
0.359 ± 0.012
0.213 ± 0.019
0.066 ± 0.019
0.281 ± 0.034
0.244 ± 0.016
0.155 ± 0.023
0.243 ± 0.010
Average precision ↑
LLRD
0.506 ± 0.018
0.819 ± 0.031
0.905 ± 0.020
0.421 ± 0.033
0.841 ± 0.009
0.934 ± 0.010
0.775 ± 0.008
MLFE
0.490 ± 0.017
0.812 ± 0.032
0.901 ± 0.021
0.410 ± 0.029
0.835 ± 0.019
0.928 ± 0.013
0.766 ± 0.016
LIFT
0.502 ± 0.021
0.824 ± 0.024
0.880 ± 0.030
0.416 ± 0.031
0.820 ± 0.018
0.922 ± 0.008
0.768 ± 0.018
RELIAB
0.497 ± 0.016
0.801 ± 0.021
0.869 ± 0.020
0.405 ± 0.041
0.781 ± 0.009
0.851 ± 0.008
0.751 ± 0.010
ML2
0.481 ± 0.030
0.816 ± 0.031
0.898 ± 0.022
0.404 ± 0.031
0.832 ± 0.014
0.930 ± 0.009
0.759 ± 0.020
CLR
0.425 ± 0.034
0.770 ± 0.019
0.695 ± 0.032
0.312 ± 0.059
0.722 ± 0.015
0.801 ± 0.012
0.755 ± 0.006
RAKEL
0.343 ± 0.009
0.772 ± 0.037
0.798 ± 0.018
0.228 ± 0.020
0.731 ± 0.017
0.777 ± 0.023
0.717 ± 0.007
MacroaveragingF1 ↑
LLRD
0.231 ± 0.026
0.676 ± 0.051
0.736 ± 0.050
0.408 ± 0.028
0.666 ± 0.024
0.800 ± 0.016
0.420 ± 0.030
MLFE
0.239 ± 0.025
0.668 ± 0.050
0.702 ± 0.056
0.415 ± 0.041
0.655 ± 0.021
0.787 ± 0.015
0.430 ± 0.024
LIFT
0.179 ± 0.014
0.651 ± 0.035
0.694 ± 0.052
0.392 ± 0.045
0.624 ± 0.033
0.788 ± 0.018
0.377 ± 0.019
RELIAB
0.288 ± 0.015
0.639 ± 0.038
0.686 ± 0.058
0.394 ± 0.031
0.568 ± 0.030
0.671 ± 0.021
0.409 ± 0.023
ML2
0.226 ± 0.024
0.656 ± 0.045
0.686 ± 0.058
0.382 ± 0.035
0.652 ± 0.018
0.783 ± 0.015
0.438 ± 0.017
CLR
0.220 ± 0.017
0.604 ± 0.032
0.616 ± 0.118
0.402 ± 0.056
0.523 ± 0.027
0.635 ± 0.013
0.386 ± 0.016
RAKEL
0.195 ± 0.010
0.615 ± 0.030
0.679 ± 0.037
0.377 ± 0.054
0.545 ± 0.018
0.654 ± 0.012
0.441 ± 0.011
MicroaveragingF1 ↑
LLRD
0.325 ± 0.011
0.692 ± 0.048
0.814 ± 0.030
0.126 ± 0.027
0.665 ± 0.024
0.792 ± 0.017
0.656 ± 0.011
MLFE
0.384 ± 0.017
0.683 ± 0.047
0.785 ± 0.031
0.137 ± 0.032
0.653 ± 0.024
0.781 ± 0.015
0.643 ± 0.013
LIFT
0.313 ± 0.013
0.664 ± 0.015
0.763 ± 0.031
0.168 ± 0.034
0.625 ± 0.031
0.779 ± 0.022
0.650 ± 0.016
RELIAB
0.454 ± 0.011
0.647 ± 0.038
0.748 ± 0.024
0.188 ± 0.028
0.562 ± 0.021
0.639 ± 0.013
0.631 ± 0.015
ML2
0.366 ± 0.013
0.674 ± 0.042
0.780 ± 0.021
0.074 ± 0.031
0.650 ± 0.019
0.776 ± 0.018
0.635 ± 0.018
CLR
0.330 ± 0.012
0.626 ± 0.029
0.606 ± 0.143
0.165 ± 0.050
0.531 ± 0.008
0.634 ± 0.017
0.623 ± 0.010
RAKEL
0.356 ± 0.025
0.648 ± 0.024
0.669 ± 0.016
0.155 ± 0.019
0.533 ± 0.005
0.645 ± 0.009
0.637 ± 0.011Table 3
Performance of each multilabel algorithm (mean ± std. deviation) on the large-scale datasets.
Comparing algorithms
Slashdot
rcv1subset1
rcv1subset2
rcv1subset3
rcv1subset4
rcv1subset5
One-error ↓
LLRD
0.363 ± 0.026
0.414 ± 0.013
0.411 ± 0.017
0.416 ± 0.029
0.317 ± 0.015
0.401 ± 0.018
MLFE
0.374 ± 0.027
0.406 ± 0.018
0.399 ± 0.013
0.402 ± 0.025
0.328 ± 0.013
0.392 ± 0.008
LIFT
0.393 ± 0.033
0.427 ± 0.011
0.434 ± 0.017
0.441 ± 0.020
0.363 ± 0.019
0.430 ± 0.019
RELIAB
0.508 ± 0.022
0.449 ± 0.015
0.458 ± 0.028
0.454 ± 0.012
0.433 ± 0.024
0.423 ± 0.009
ML2
0.370 ± 0.025
0.404 ± 0.017
0.395 ± 0.018
0.398 ± 0.021
0.323 ± 0.021
0.388 ± 0.010
CLR
0.965 ± 0.013
0.513 ± 0.022
0.515 ± 0.009
0.518 ± 0.028
0.472 ± 0.031
0.521 ± 0.021
RAKEL
0.602 ± 0.009
0.605 ± 0.013
0.574 ± 0.012
0.585 ± 0.022
0.561 ± 0.022
0.614 ± 0.009
Coverage ↓
LLRD
0.107 ± 0.010
0.125 ± 0.008
0.121 ± 0.009
0.123 ± 0.006
0.092 ± 0.004
0.116 ± 0.009
MLFE
0.126 ± 0.013
0.136 ± 0.005
0.130 ± 0.010
0.129 ± 0.007
0.094 ± 0.007
0.124 ± 0.007
LIFT
0.112 ± 0.008
0.144 ± 0.020
0.135 ± 0.008
0.156 ± 0.008
0.113 ± 0.012
0.148 ± 0.013
RELIAB
0.131 ± 0.007
0.152 ± 0.012
0.128 ± 0.014
0.144 ± 0.011
0.105 ± 0.020
0.131 ± 0.014
ML2
0.103 ± 0.011
0.138 ± 0.008
0.132 ± 0.010
0.126 ± 0.006
0.078 ± 0.006
0.129 ± 0.009
CLR
0.254 ± 0.003
0.146 ± 0.018
0.141 ± 0.007
0.137 ± 0.010
0.109 ± 0.018
0.136 ± 0.011
RAKEL
0.226 ± 0.020
0.426 ± 0.023
0.372 ± 0.016
0.381 ± 0.014
0.365 ± 0.009
0.388 ± 0.020
Ranking loss ↓
LLRD
0.090 ± 0.010
0.049 ± 0.004
0.050 ± 0.004
0.052 ± 0.002
0.038 ± 0.002
0.047 ± 0.003
MLFE
0.107 ± 0.013
0.052 ± 0.002
0.055 ± 0.007
0.055 ± 0.002
0.040 ± 0.004
0.050 ± 0.003
LIFT
0.098 ± 0.016
0.058 ± 0.007
0.057 ± 0.009
0.068 ± 0.004
0.059 ± 0.010
0.055 ± 0.007
RELIAB
0.124 ± 0.003
0.066 ± 0.010
0.063 ± 0.008
0.062 ± 0.004
0.052 ± 0.006
0.063 ± 0.005
ML2
0.103 ± 0.012
0.056 ± 0.004
0.057 ± 0.004
0.056 ± 0.003
0.031 ± 0.003
0.050 ± 0.004
CLR
0.237 ± 0.008
0.062 ± 0.011
0.066 ± 0.008
0.065 ± 0.012
0.047 ± 0.006
0.071 ± 0.005
RAKEL
0.211 ± 0.019
0.226 ± 0.019
0.215 ± 0.017
0.230 ± 0.015
0.235 ± 0.014
0.214 ± 0.016
Average precision ↑
LLRD
0.725 ± 0.019
0.611 ± 0.010
0.638 ± 0.011
0.634 ± 0.017
0.717 ± 0.008
0.643 ± 0.011
MLFE
0.712 ± 0.021
0.618 ± 0.016
0.645 ± 0.009
0.639 ± 0.014
0.708 ± 0.012
0.647 ± 0.012
LIFT
0.703 ± 0.010
0.586 ± 0.009
0.598 ± 0.012
0.595 ± 0.011
0.674 ± 0.013
0.598 ± 0.011
RELIAB
0.624 ± 0.014
0.578 ± 0.021
0.611 ± 0.011
0.614 ± 0.018
0.655 ± 0.018
0.604 ± 0.009
ML2
0.715 ± 0.022
0.621 ± 0.012
0.647 ± 0.013
0.643 ± 0.016
0.717 ± 0.013
0.650 ± 0.010
CLR
0.269 ± 0.002
0.575 ± 0.013
0.584 ± 0.021
0.571 ± 0.032
0.614 ± 0.020
0.588 ± 0.013
RAKEL
0.522 ± 0.020
0.395 ± 0.012
0.445 ± 0.018
0.431 ± 0.014
0.450 ± 0.012
0.437 ± 0.016
MacroaveragingF1 ↑
LLRD
0.427 ± 0.035
0.235 ± 0.020
0.259 ± 0.019
0.213 ± 0.031
0.300 ± 0.019
0.211 ± 0.020
MLFE
0.466 ± 0.035
0.198 ± 0.017
0.195 ± 0.056
0.202 ± 0.030
0.249 ± 0.021
0.204 ± 0.021
LIFT
0.429 ± 0.037
0.223 ± 0.025
0.186 ± 0.024
0.200 ± 0.031
0.238 ± 0.013
0.196 ± 0.031
RELIAB
0.425 ± 0.029
0.342 ± 0.022
0.338 ± 0.016
0.348 ± 0.014
0.342 ± 0.028
0.352 ± 0.014
ML2
0.472 ± 0.029
0.216 ± 0.020
0.206 ± 0.024
0.195 ± 0.030
0.244 ± 0.023
0.208 ± 0.011
CLR
0.174 ± 0.032
0.285 ± 0.032
0.264 ± 0.021
0.272 ± 0.022
0.311 ± 0.031
0.305 ± 0.017
RAKEL
0.354 ± 0.037
0.269 ± 0.030
0.251 ± 0.014
0.255 ± 0.014
0.263 ± 0.014
0.274 ± 0.018
MicroaveragingF1 ↑
LLRD
0.496 ± 0.021
0.393 ± 0.013
0.381 ± 0.017
0.406 ± 0.027
0.470 ± 0.013
0.402 ± 0.018
MLFE
0.545 ± 0.019
0.373 ± 0.014
0.375 ± 0.031
0.392 ± 0.024
0.403 ± 0.020
0.381 ± 0.017
LIFT
0.510 ± 0.030
0.320 ± 0.017
0.353 ± 0.014
0.347 ± 0.018
0.342 ± 0.024
0.363 ± 0.008
RELIAB
0.453 ± 0.011
0.408 ± 0.010
0.449 ± 0.008
0.451 ± 0.021
0.478 ± 0.016
0.454 ± 0.012
ML2
0.556 ± 0.022
0.371 ± 0.014
0.391 ± 0.010
0.383 ± 0.026
0.393 ± 0.022
0.410 ± 0.015
CLR
0.104 ± 0.005
0.367 ± 0.011
0.368 ± 0.024
0.320 ± 0.024
0.381 ± 0.015
0.372 ± 0.008
RAKEL
0.365 ± 0.020
0.359 ± 0.023
0.348 ± 0.016
0.341 ± 0.016
0.371 ± 0.015
0.342 ± 0.006We useFriedman test [38] based on the average ranks for verifying whether the difference between algorithms is statistically significant. If the assumption that “all algorithms have equal performance” is rejected, it means that the performance of each algorithm is significantly different. As can be seen from the data presented in Table 4, the hypothesis that there is no significant difference among the algorithms is not valid under the condition of 0.05 significance level. Therefore, we need to conduct a post hoc test to further distinguish the various algorithms. Usually, there are two options for post hoc test, one is the Nemenyi test [38] and the other is the Bonferroni–Dunn test [39]. For k algorithms, the former needs to compare kk−1/2 times, while the latter only needs k−1 times in some cases. Thus, we choose the latter. The Bonferroni–Dunn test is used to test whether LLRD is more competitive than the comparative algorithm, in which LLRD plays a role of control algorithm. When the difference of average rank between two algorithms is more than one critical difference CD, the performance of two algorithms is obviously different. The CD value mentioned here can be calculated from CD=qαkk+1/6N, where k = 7 and N = 13, when the significance level is 0.05, the corresponding qα=2.638.Table 4
The Friedman statisticsFF and the critical value.
Evaluation metric
F
F
Critical value
One-error
34.0909
2.2274
Coverage
20.3765
Ranking loss
21.1642
Average precision
39.8409
Macroaveraging F1
2.6520
Microaveraging F1
7.6088The CD diagram associated with LLRD and its comparison algorithm is shown in Figure1. The numbers on the horizontal axis of the coordinate indicate the average rank value of each algorithm under different evaluation criteria. There is no significant difference in performance among the various algorithms connected by solid lines.Figure 1
Comparison of LLRD (control algorithm) against other related approaches with the Bonferroni–Dunn test. Approaches that are not connected to LLRD are significantly different in performance from LLRD. (a) One-error. (b) Coverage. (c) Ranking loss. (d) Average precision. (e) MacroaveragingF1. (f) Microaveraging F2.
(a)
(b)
(c)
(d)
(e)
(f)Through the analysis of the above experimental results, we can draw the following conclusions:(1)
In terms of the four evaluation criteria ofone-error, coverage, ranking loss, and average precision, LLRD is obviously superior to RELIAB, RAKEL, and CLR.(2)
The smaller the average rank value, the better the performance of the corresponding. For LLRD, five of the average rank value in the six CD subdiagrams are optimal, which shows LLRD outperforms other algorithms.(3)
For regular-size datasets, LLRD ranks first in 69% of the cases under different evaluation criteria, while for large-scale datasets, it ranks first in 36.1%.


## 4. Conclusions

In this work, we propose a novel multilabel classification algorithm named LLRD, which adopts the low-rank decomposition to gain the internal information of label and further reduce the information loss of the label transformation via the new feature space. Experimental results show that the performance of the proposed LLRD is better than many state-of-the-art multilabel classification techniques. In the future, we will explore alternative models combining the low-rank decomposition and classification into a joint optimization problem for considering more complex correlation of labels.



---

*Source: 1279253-2020-04-07.xml*