# Reordering Features with Weights Fusion in Multiclass and Multiple-Kernel Speech Emotion Recognition

Authors: Xiaoqing Jiang; Kewen Xia; Lingyin Wang; Yongliang Lin
Journal: Journal of Electrical and Computer Engineering
 (2017)

---

## Abstract

The selection of feature subset is a crucial aspect in speech emotion recognition problem. In this paper, a Reordering Features with Weights Fusion (RFWF) algorithm is proposed for selecting more effective and compact feature subset. The RFWF algorithm fuses the weights reflecting the relevance, complementarity, and redundancy between features and classes comprehensively and implements the reordering of features to construct feature subset with excellent emotional recognizability. A binary-tree structured multiple-kernel SVM classifier is adopted in emotion recognition. And different feature subsets are selected in different nodes of the classifier. The highest recognition accuracy of the five emotions in Berlin database is 90.549% with only 15 features selected by RFWF. The experimental results show the effectiveness of RFWF in building feature subset and the utilization of different feature subsets for specified emotions can improve the overall recognition performance.


---

## Body

## 1. Introduction

Feature selection is a crucial aspect in pattern recognition problems. In multiclass SVM classifier, for example, the structure of the classifier can be one-to-one, one-to-all, hierarchy, or tree structure, so several SVM nodes or models exist in the multiclass classifier [1–3]. There are two questions in speech emotion recognition (SER): (1) how to seek the optimal feature subset from the acoustic features; (2) whether the same acoustic feature subset is proper in all nodes of the multiclass classifier. These questions are researched in this paper. A novel algorithm named Reordering Features with Weights Fusion (RFWF) is proposed to select feature subsets. And in emotion recognition procedure, different feature subsets are adopted in SVM nodes to recognize different emotions.In SER field, the dimension of feature set ranges from tens to hundreds. However, the increasing dimension does not mean a radical improvement of the recognition accuracy, because the variety and redundancy between more and more features influence the overall performance and complexity of the system [4]. And there is not a categorical assertion about the most effective feature set in SER nowadays. Feature selection algorithms used in machine learning widely can choose the optimal feature subset with the least generalization error. There are three types of feature selection methods: the wrapper method, the embedded method, and the filter method [5]. Compared with the wrapper method and the embedded method, the filter method is simpler and faster in calculation and its learning strategy is more robust to overfitting. Additionally, because the selection result of the filter method is independent of the learning model, the filter method can be adopted in a variety of leaning tasks. The criteria in filter methods mainly focus on the relevance, the redundancy, and the complementarity. For example, Joint Mutual Information (JMI) [6] considers the relevance between features and classes. Fast correlation-based filter (FCBF) [7] takes the redundancy between features into account. Max-Relevance Min-Redundancy (MRMR) [8] gives consideration to both relevance and redundancy to find the balance between the two properties. In Conditional Information Feature Extraction (CIFE) [9], the information provided by the features is divided into two parts: the class-relevant information that benefits the classification and the class-redundant information that disturbs the classification. And the key idea of Double Input Symmetrical Relevance (DISR) [10] is the utilization of symmetric relevance to consider the complementarity between two input features. In the SER field, various feature selection criteria are often adopted [7, 11–13], and different criterion emphasizes different aspects. Reordering Features with Weights Fusion (RFWF) algorithm proposed in this paper aims to consider relevance, redundancy, and complementarity comprehensively.Traditionally the same feature subset is adopted in all emotional classes for training and testing [14]. In [11], different feature subsets are adopted on two emotional speech databases, but the emotional recognizability of the features to different emotions has not been considered. Research has shown that acoustic features have different recognizability to specific emotions. For example, pitch related features are usually essential to classify happy and sad [15], while they are often weak in the recognition between happy and surprise because of their high values in these emotions [16]. In order to improve the performance of the whole system, different feature subsets are selected and adopted on the different nodes of the multiclass classifier in this paper.The content of the paper is arranged as follows: Section2 gives the basic concepts of filter feature selection and the method of RFWF; Section 3 introduces the structure of the multiclass and multiple-kernel SVM classifier; Section 4 is the analysis of experiments including the results of RFWF and recognition accuracies of emotions; and the final section is devoted to the conclusions.


## 2. Features and Feature Selection Methods

### 2.1. Acoustic Features

Speech acoustic features usually used in SER are the prosodic features, voice quality features, and spectral features. In this paper, 409 utterances in Berlin database [17] of 5 emotions including happy (71 samples), angry (127 samples), fear (69 samples), sad (63 samples), and neutral (79 samples) are studied. These samples are separated into training and testing categories randomly. The training samples are 207 including happy (36 samples), angry (64 samples), fear (35 samples), sad (32 samples), and neutral (40 samples), and the rest 202 ones are the test samples.Pitch, energy, time, formant, and Mel Frequency Cepstrum Coefficient (MFCC) and their statistics parameters are extracted. The total dimension the feature set is 45. Table1 lists the acoustic features and their sequence indices in this paper.Table 1
Feature set.
Type
Feature
Statistic parameters
Prosodic feature
Pitch
Maximum (1), minimum (2), range (3), mean (4), Std (5), first quartile (6), median (7), third quartile (8), interquartile range (9)
Energy
Maximum (10), minimum (11), range (12), mean (13), Std (14), first quartile (15), median (16), third quartile (17), interquartile range (18)
Time
Total frames (19), voiced frames (20), unvoiced frames (21), ratio of voiced frames versus unvoiced frames (22), ratio of voiced frames versus total frames (23), ratio of unvoiced frames versus total frames (24)
Voice quality feature
Formant
F
1:mean (25), Std (26), median (27)
F
2: mean (28), Std (29), median (30)
F
3: mean (31), Std (32), median (33)
Spectral feature
MFCC
12 MFCC (34–45)


### 2.2. Mathematical Description of Feature Selection

Relevance, redundancy, and complementarity are considered in feature selection methods. If a feature can provide information about the class, the relevance exists between the feature and class. The redundancy is based on the dependency between the selected and unselected features. And complementarity means that the interaction between an individual feature and the selected feature subset is beneficial to the classification. Complementarity is important in the cases of null relevance, such as XOR problem [10, 18].The concepts of information theory, such as mutual information denoted byI and entropy denoted by H, are widely used in feature selection. Mathematically, Fii=1,…,409 is the feature vector of ith sample, and the fi,j is the jth feature of the ith sample in the feature set F. The selected subset and unselected subset are Fs and F-s with the mathematic relation of Fs∩F-s=⌀ and Fs∪F-s=F. Cn, n=1,…,5 is the specified emotion in Berlin database. In the following content, the mathematical description of relevance, redundancy, and complementarity is interpreted through the introduction of MRMR and DISR.In MRMRFs={fp}, p=1,…,d and fq∈F-s, q=1,…,45-d, the relevance term uq=I(fq;C) and the redundancy term zq=1/d∑fp∈FsI(fp;fq) are used in the criterion:(1)fqMRMR=argmaxfq∈F-suq-zq,where I(fq;C) can represent the relevance between an unselected feature and the class and I(fp;fq) can represent the redundancy between the unselected and selected features. The detailed computation can be found in [8].The key idea of DISR depends on the consideration of the second average sub-subset information criterion in (2) to consider the complementarity between an unselected feature fq and a selected feature fp given the specific classC. (2)fqDISR=argmaxfq∈F-s∑fp∈FsIfp,q;C.Equation (2) also can be modified by a normalized relevance measure named symmetric relevance calculated by the following:(3)SR=Ifp,q;CHfp,q;C.In DISR,Ifp,q;C is the complementarity calculated by(4)Ifp,q;C=Ifp;C+Ifq;C-Afp;fq;C,where A(fp;fq;C) stands for the interaction among fp, fq, andC. From its general meaning, for n sets of random variables X1,X2,…,Xn, the interaction can be defined as(5)AX1,X2,…,Xn=∑k=1n∑S⊆1,…,n:S=k-1k+1HXS.The detailed definition and proof can be found in [10].


### 2.3. Reordering Features with Weights Fusion

For the comprehensive consideration of relevance, redundancy, and complementarity, the following criterion named Reordering Features with Weights Fusion (RFWF) is proposed to fuse the intrinsic properties of the features:(6)fqRFWF=argmaxfq∈F-sW1Ifq;C+W2∑fp∈FsIfp,q;C-W3∑fp∈FsIfp;fq,where W1, W2, and W3 are the fusing weights of the unselected feature fq and they are combined in (6) to reflect the contribution of fq to given class. The procedure of RFWF algorithm described in the following is illustrated in Figure 1:(1)
Lm(fq)(m=1,2,3) is the sequence number of the feature fq ranked in order of the values of I(fq;C), ∑fp∈FsIfp,q;C and ∑fp∈FsIfp;fq, respectively. If the dimension of the feature set is 45, Lmfq is an integer value ranging within 1~45. For example, if the Ifq;C is the largest, L1(fq) is 1. And if the ∑fp∈FsI(fp,q;C) is the lowest, L2fq is 45. The initial selected feature fp in Fs is confirmed by the largest value of Ifp;C.(2)
Weighted values can be calculated by the following formula:(7)Wmfq=45-Lmfq+1,m=1,2,3,q=1,…,45.For example, if L1fq is 1, the corresponding weight about the relevance between feature and class is W1(I(fq;C))=45.(3)
All of the features can be reordered by the fusing result usingW1, W2, and W3.(4)
The topN features can be selected to construct the optimal feature subset.Figure 1
Procedure of RFWF algorithm.Because this algorithm fuses three weights to consider the contribution of features in the classification and a reordering process exists in the process, the algorithm is named Reordering Features with Weights Fusion (RFWF).


## 2.1. Acoustic Features

Speech acoustic features usually used in SER are the prosodic features, voice quality features, and spectral features. In this paper, 409 utterances in Berlin database [17] of 5 emotions including happy (71 samples), angry (127 samples), fear (69 samples), sad (63 samples), and neutral (79 samples) are studied. These samples are separated into training and testing categories randomly. The training samples are 207 including happy (36 samples), angry (64 samples), fear (35 samples), sad (32 samples), and neutral (40 samples), and the rest 202 ones are the test samples.Pitch, energy, time, formant, and Mel Frequency Cepstrum Coefficient (MFCC) and their statistics parameters are extracted. The total dimension the feature set is 45. Table1 lists the acoustic features and their sequence indices in this paper.Table 1
Feature set.
Type
Feature
Statistic parameters
Prosodic feature
Pitch
Maximum (1), minimum (2), range (3), mean (4), Std (5), first quartile (6), median (7), third quartile (8), interquartile range (9)
Energy
Maximum (10), minimum (11), range (12), mean (13), Std (14), first quartile (15), median (16), third quartile (17), interquartile range (18)
Time
Total frames (19), voiced frames (20), unvoiced frames (21), ratio of voiced frames versus unvoiced frames (22), ratio of voiced frames versus total frames (23), ratio of unvoiced frames versus total frames (24)
Voice quality feature
Formant
F
1:mean (25), Std (26), median (27)
F
2: mean (28), Std (29), median (30)
F
3: mean (31), Std (32), median (33)
Spectral feature
MFCC
12 MFCC (34–45)


## 2.2. Mathematical Description of Feature Selection

Relevance, redundancy, and complementarity are considered in feature selection methods. If a feature can provide information about the class, the relevance exists between the feature and class. The redundancy is based on the dependency between the selected and unselected features. And complementarity means that the interaction between an individual feature and the selected feature subset is beneficial to the classification. Complementarity is important in the cases of null relevance, such as XOR problem [10, 18].The concepts of information theory, such as mutual information denoted byI and entropy denoted by H, are widely used in feature selection. Mathematically, Fii=1,…,409 is the feature vector of ith sample, and the fi,j is the jth feature of the ith sample in the feature set F. The selected subset and unselected subset are Fs and F-s with the mathematic relation of Fs∩F-s=⌀ and Fs∪F-s=F. Cn, n=1,…,5 is the specified emotion in Berlin database. In the following content, the mathematical description of relevance, redundancy, and complementarity is interpreted through the introduction of MRMR and DISR.In MRMRFs={fp}, p=1,…,d and fq∈F-s, q=1,…,45-d, the relevance term uq=I(fq;C) and the redundancy term zq=1/d∑fp∈FsI(fp;fq) are used in the criterion:(1)fqMRMR=argmaxfq∈F-suq-zq,where I(fq;C) can represent the relevance between an unselected feature and the class and I(fp;fq) can represent the redundancy between the unselected and selected features. The detailed computation can be found in [8].The key idea of DISR depends on the consideration of the second average sub-subset information criterion in (2) to consider the complementarity between an unselected feature fq and a selected feature fp given the specific classC. (2)fqDISR=argmaxfq∈F-s∑fp∈FsIfp,q;C.Equation (2) also can be modified by a normalized relevance measure named symmetric relevance calculated by the following:(3)SR=Ifp,q;CHfp,q;C.In DISR,Ifp,q;C is the complementarity calculated by(4)Ifp,q;C=Ifp;C+Ifq;C-Afp;fq;C,where A(fp;fq;C) stands for the interaction among fp, fq, andC. From its general meaning, for n sets of random variables X1,X2,…,Xn, the interaction can be defined as(5)AX1,X2,…,Xn=∑k=1n∑S⊆1,…,n:S=k-1k+1HXS.The detailed definition and proof can be found in [10].


## 2.3. Reordering Features with Weights Fusion

For the comprehensive consideration of relevance, redundancy, and complementarity, the following criterion named Reordering Features with Weights Fusion (RFWF) is proposed to fuse the intrinsic properties of the features:(6)fqRFWF=argmaxfq∈F-sW1Ifq;C+W2∑fp∈FsIfp,q;C-W3∑fp∈FsIfp;fq,where W1, W2, and W3 are the fusing weights of the unselected feature fq and they are combined in (6) to reflect the contribution of fq to given class. The procedure of RFWF algorithm described in the following is illustrated in Figure 1:(1)
Lm(fq)(m=1,2,3) is the sequence number of the feature fq ranked in order of the values of I(fq;C), ∑fp∈FsIfp,q;C and ∑fp∈FsIfp;fq, respectively. If the dimension of the feature set is 45, Lmfq is an integer value ranging within 1~45. For example, if the Ifq;C is the largest, L1(fq) is 1. And if the ∑fp∈FsI(fp,q;C) is the lowest, L2fq is 45. The initial selected feature fp in Fs is confirmed by the largest value of Ifp;C.(2)
Weighted values can be calculated by the following formula:(7)Wmfq=45-Lmfq+1,m=1,2,3,q=1,…,45.For example, if L1fq is 1, the corresponding weight about the relevance between feature and class is W1(I(fq;C))=45.(3)
All of the features can be reordered by the fusing result usingW1, W2, and W3.(4)
The topN features can be selected to construct the optimal feature subset.Figure 1
Procedure of RFWF algorithm.Because this algorithm fuses three weights to consider the contribution of features in the classification and a reordering process exists in the process, the algorithm is named Reordering Features with Weights Fusion (RFWF).


## 3. Multiclass and Multiple-Kernel SVM Classifier with Binary-Tree Structure

The Support Vector Machine (SVM) is a discriminative classifier proposed for binary classification problem and based on the theory of structural risk minimization. The performance of a single-kernel method depends heavily on the choice of the kernel. If a dataset has varying distributions, a single kernel may not be adequate. Kernel fusion has been proposed to deal with this problem [19].The simplest kernel fusion is a weighted combination ofM kernels:(8)K=∑s=1MμsKs,where μs is the optimal weights and Ks is thesth kernel matrix. The selection of μs is an optimal question, and the objective function and constraints of the problem can be formulated as the Semidefinite Programming (SDP) form. The detailed proof of can be found in [20].In this paper, a multiple-kernel SVM classifier with an unbalance binary-tree structure illustrated in Figure2 is adopted. In Figure 2, there are five emotions to be recognized. The first classifying node (Model 1) is improved by multiple-kernel SVM to recognize the most confusable emotion while the subsequent classifying nodes still retain single-kernel SVM. This arrangement attributes the reduction of recognition error accumulation and the computing complexity required in the calculation of the multiple-kernel matrices for all nodes.Figure 2
The binary-tree structured multiclass and multiple-kernel SVM classifier.According to the previous works [2, 11, 21, 22], happy is the most confusable emotion and its recognition accuracy is the main factor influencing the total performance in Berlin database. Thus in the classifier shown in Figure 2, happy is 1, angry is 2, fear is 3, neutral is 4, and sad is 5. Feature subset selected by RFWF is adopted in the SVM training and testing, where Model 1 is learned by multiple kernels. Models 2, 3, and 4 are still single-kernel SVM models.


## 4. Experiments and Analysis

### 4.1. The Experimental Results of RFWF

Table2 lists the reordering results of features for the four SVM models in Figure 2 according to the fusing results of W1, W2, and W3. In Table 2, the numbers are the indices of the features listed in Table 1.Table 2
RFWF results of the feature set in different models of the classifier.
Indices of features
Model 1
5
37
38
42
8
1
9
20
40
33
26
3
4
45
18
44
15
24
21
13
29
10
6
7
14
16
19
23
17
35
41
22
34
25
2
12
31
43
36
39
28
30
32
11
27
Model 2
36
34
20
43
42
7
5
14
18
41
16
8
4
15
1
6
33
37
24
19
23
45
9
3
31
13
17
21
10
32
22
39
40
2
30
28
35
12
38
11
29
27
44
26
25
Model 3
4
20
43
35
36
45
8
16
7
18
9
23
34
11
25
19
21
2
6
5
22
44
27
10
24
38
32
29
42
12
26
40
14
17
39
13
31
15
37
41
28
30
1
33
3
Model 4
23
7
39
41
14
10
15
19
24
22
13
21
40
35
11
16
42
25
27
38
8
18
33
36
43
12
20
6
29
5
30
17
31
4
2
37
28
9
44
1
26
34
3
32
45It is clear that, in the four SVM models, the contribution of different features to emotional recognizability is distinct. For example, the standard deviation of pitch (feature sequence index is 5) is the most essential feature to classify happy and the other emotions in Berlin database, while ratio of voiced frames versus total frames (feature sequence index is 23) is the most important feature to recognize neutral and sad. The results show that it is necessary to adopt different feature subsets to recognize different emotions.


### 4.2. Experimental Results of SER and Analysis

In the SER experiments, LibSVM package is adopted. Three basis Radial Basis Functions (RBF) kernel functions with parameters ofγ1=0.01, γ2=0.1, γ3=1 are combined in Model 1. YALMIP toolbox is used to solve the SDP problem and find three μs with the features listed in Table 1. In single-kernel SVM models, the value of γ is 1/k, where k is the selected feature number in the recognition procedure. When the selected feature number is specified, the same γ is adopted for all single-kernel models.Recognition accuracies, Root Mean Square Error (RMSE) and Maximum Error (MaxE) are used to evaluate the performance of the SVM classifier. RMSE and MaxE are calculated by following equations:(9)PRMSE=15∑i=15ei2QMaxE=max⁡ei,where ei is the recognition error (%) ofith emotion. Obviously, the higher the recognition accuracies and the lower the values of RMSE and MaxE, the better the performance of the classifier.If the dimension of feature subset isN, then the topN features in the Table 2 are selected to construct the feature subset. N ranging within 1–45 achieves different recognition performance. Figure 3 plots the curves of total emotion recognition accuracies of MRMR, DISR, and RFWF features selection algorithms, respectively, where RFWF is adopted in multiple-kernel (RFWF MK) and single-kernel (RFWF SK) SVM classifiers. Different feature subsets are selected for Models 1–4. Figure 4 gives the RMSE and MaxE corresponding to Figure 3. In Figures 3 and 4, the horizontal axis describes the different number of the selected features or the dimension of the selected feature subset. Table 3 lists the detailed experimental data of the highest total accuracies of MRMR, DISR, RFWF MK, and RFWF SK methods.Table 3
Emotion recognition accuracies of different feature selection methods with the best feature number.
Selection methods
Feature number
SER accuracies (%)
Total
Angry
Fear
Happy
Neutral
Sad
DISR
39
88.614
93.651
79.412
97.143
74.359
96.774
MRMR
39
90.099
92.063
67.647
97.143
94.872
96.774
RFWF MK
15
90.594
90.476
79.412
97.143
87.179
100
RFWF SK
39
79.208
95.238
73.529
28.571
87.179
100Figure 3
Emotional recognition accuracies.Figure 4
RMSE and MaxE.The recognition results show DISR and MRMR algorithms reach their highest accuracies with 39 features. However, the highest accuracy of RFWF MK is 90.594% with only 15 features. The accuracies of DISR and MRMR are 70.792% and 77.723%, respectively, when the selected feature number is 15. When the selected feature number is 45, it means that no feature selection algorithms are utilized. In this situation, the performances of DISR, MRMR, and RFWF MK are the same and the total accuracy is 83.663%. These results show that the RFWF algorithm has the best performance with the lowest dimension of feature subset. The corresponding RMSE and MaxE curves of RFWF are the lowest when the selected feature numberN is below 30. If the dimension of the feature subset increases, the three feature selection methods with multiple-kernel classifier have similar performance. This is mainly because RFWF uses the same weighing methods for the relevance, redundancy, and complementarity. From this aspect, it is an average strategy in the procedure of weights fusion. The results show that when the dimension feature subset is close to 45, the RFWF degrades to handle with the complex inherent properties between the features and more optimal feature fusion method should be studied.The highest total accuracy of RFWF SK is 79.208%, which is much lower than accuracy of RFWF MK. The recognition accuracies of happy are 97.143% steadily in the three methods when Model 1 is improved by multiple-kernel SVM. The experimental results demonstrate that multiple-kernel classifier can solve the confusion between happy and other emotions effectively, which cannot be dealt with by single-kernel SVM. The highest SER accuracies of RFWF MK can be compared to the results of the Enhanced Sparse Representation Classifier (Enhanced-SRC) in [11] and feature fusion based on MKL in [23]. The experimental comparison is listed in Table 4, where the symbol “N” denotes no relating experimental results in the reference.Table 4
Comparison on SER accuracies (%) of 5 emotions in Berlin database.
Methods
Anger
Fear
Happy
Neutral
Sad
Enhanced-SRC
98.55
83.16
57.73
70.08
96.71
Feature fusion based on MKL
81
83
N
65
95
RFWF MK
90.476
79.412
97.143
87.179
100If Models 2–4 use the same feature subset as in Model 1 with the dimension of 15, the accuracy of RFWF MK is only 63.861%. And when all models use the same feature subset of 39, the highest accuracy of RFWF MK is 85.149%. The data confirms that the utilization of the same feature subset in all models influences the emotion recognition performance negatively. These experimental results demonstrate that different feature subset is necessary in the recognition of different emotions, which also indicates the difficulty to build a robust and effective feature subset for all emotions.


## 4.1. The Experimental Results of RFWF

Table2 lists the reordering results of features for the four SVM models in Figure 2 according to the fusing results of W1, W2, and W3. In Table 2, the numbers are the indices of the features listed in Table 1.Table 2
RFWF results of the feature set in different models of the classifier.
Indices of features
Model 1
5
37
38
42
8
1
9
20
40
33
26
3
4
45
18
44
15
24
21
13
29
10
6
7
14
16
19
23
17
35
41
22
34
25
2
12
31
43
36
39
28
30
32
11
27
Model 2
36
34
20
43
42
7
5
14
18
41
16
8
4
15
1
6
33
37
24
19
23
45
9
3
31
13
17
21
10
32
22
39
40
2
30
28
35
12
38
11
29
27
44
26
25
Model 3
4
20
43
35
36
45
8
16
7
18
9
23
34
11
25
19
21
2
6
5
22
44
27
10
24
38
32
29
42
12
26
40
14
17
39
13
31
15
37
41
28
30
1
33
3
Model 4
23
7
39
41
14
10
15
19
24
22
13
21
40
35
11
16
42
25
27
38
8
18
33
36
43
12
20
6
29
5
30
17
31
4
2
37
28
9
44
1
26
34
3
32
45It is clear that, in the four SVM models, the contribution of different features to emotional recognizability is distinct. For example, the standard deviation of pitch (feature sequence index is 5) is the most essential feature to classify happy and the other emotions in Berlin database, while ratio of voiced frames versus total frames (feature sequence index is 23) is the most important feature to recognize neutral and sad. The results show that it is necessary to adopt different feature subsets to recognize different emotions.


## 4.2. Experimental Results of SER and Analysis

In the SER experiments, LibSVM package is adopted. Three basis Radial Basis Functions (RBF) kernel functions with parameters ofγ1=0.01, γ2=0.1, γ3=1 are combined in Model 1. YALMIP toolbox is used to solve the SDP problem and find three μs with the features listed in Table 1. In single-kernel SVM models, the value of γ is 1/k, where k is the selected feature number in the recognition procedure. When the selected feature number is specified, the same γ is adopted for all single-kernel models.Recognition accuracies, Root Mean Square Error (RMSE) and Maximum Error (MaxE) are used to evaluate the performance of the SVM classifier. RMSE and MaxE are calculated by following equations:(9)PRMSE=15∑i=15ei2QMaxE=max⁡ei,where ei is the recognition error (%) ofith emotion. Obviously, the higher the recognition accuracies and the lower the values of RMSE and MaxE, the better the performance of the classifier.If the dimension of feature subset isN, then the topN features in the Table 2 are selected to construct the feature subset. N ranging within 1–45 achieves different recognition performance. Figure 3 plots the curves of total emotion recognition accuracies of MRMR, DISR, and RFWF features selection algorithms, respectively, where RFWF is adopted in multiple-kernel (RFWF MK) and single-kernel (RFWF SK) SVM classifiers. Different feature subsets are selected for Models 1–4. Figure 4 gives the RMSE and MaxE corresponding to Figure 3. In Figures 3 and 4, the horizontal axis describes the different number of the selected features or the dimension of the selected feature subset. Table 3 lists the detailed experimental data of the highest total accuracies of MRMR, DISR, RFWF MK, and RFWF SK methods.Table 3
Emotion recognition accuracies of different feature selection methods with the best feature number.
Selection methods
Feature number
SER accuracies (%)
Total
Angry
Fear
Happy
Neutral
Sad
DISR
39
88.614
93.651
79.412
97.143
74.359
96.774
MRMR
39
90.099
92.063
67.647
97.143
94.872
96.774
RFWF MK
15
90.594
90.476
79.412
97.143
87.179
100
RFWF SK
39
79.208
95.238
73.529
28.571
87.179
100Figure 3
Emotional recognition accuracies.Figure 4
RMSE and MaxE.The recognition results show DISR and MRMR algorithms reach their highest accuracies with 39 features. However, the highest accuracy of RFWF MK is 90.594% with only 15 features. The accuracies of DISR and MRMR are 70.792% and 77.723%, respectively, when the selected feature number is 15. When the selected feature number is 45, it means that no feature selection algorithms are utilized. In this situation, the performances of DISR, MRMR, and RFWF MK are the same and the total accuracy is 83.663%. These results show that the RFWF algorithm has the best performance with the lowest dimension of feature subset. The corresponding RMSE and MaxE curves of RFWF are the lowest when the selected feature numberN is below 30. If the dimension of the feature subset increases, the three feature selection methods with multiple-kernel classifier have similar performance. This is mainly because RFWF uses the same weighing methods for the relevance, redundancy, and complementarity. From this aspect, it is an average strategy in the procedure of weights fusion. The results show that when the dimension feature subset is close to 45, the RFWF degrades to handle with the complex inherent properties between the features and more optimal feature fusion method should be studied.The highest total accuracy of RFWF SK is 79.208%, which is much lower than accuracy of RFWF MK. The recognition accuracies of happy are 97.143% steadily in the three methods when Model 1 is improved by multiple-kernel SVM. The experimental results demonstrate that multiple-kernel classifier can solve the confusion between happy and other emotions effectively, which cannot be dealt with by single-kernel SVM. The highest SER accuracies of RFWF MK can be compared to the results of the Enhanced Sparse Representation Classifier (Enhanced-SRC) in [11] and feature fusion based on MKL in [23]. The experimental comparison is listed in Table 4, where the symbol “N” denotes no relating experimental results in the reference.Table 4
Comparison on SER accuracies (%) of 5 emotions in Berlin database.
Methods
Anger
Fear
Happy
Neutral
Sad
Enhanced-SRC
98.55
83.16
57.73
70.08
96.71
Feature fusion based on MKL
81
83
N
65
95
RFWF MK
90.476
79.412
97.143
87.179
100If Models 2–4 use the same feature subset as in Model 1 with the dimension of 15, the accuracy of RFWF MK is only 63.861%. And when all models use the same feature subset of 39, the highest accuracy of RFWF MK is 85.149%. The data confirms that the utilization of the same feature subset in all models influences the emotion recognition performance negatively. These experimental results demonstrate that different feature subset is necessary in the recognition of different emotions, which also indicates the difficulty to build a robust and effective feature subset for all emotions.


## 5. Conclusions

In this paper, a RFWF feature selection method is proposed for building more effective feature subset in SER. A binary-tree structured multiclass and multiple-kernel SVM classifier is adopted to recognize emotions in a public emotional speech database. The experimental results indicate the effectiveness of the whole system.The conclusions of this paper are as follows: (1) intrinsic properties of features about relevance, redundancy, and complementarity can be considered comprehensively by weights fusion. (2) Feature subset selected by RFWF achieves higher total accuracy than MRMR and DISR with lower dimension. (3) In multiclass classifier, different feature subsets adopted in different nodes can improve the recognizability of the whole system. (4) Multiple-kernel SVM classifier is robust and effective in recognizing the most confusable emotion.The next work can focus on the research about more optimal feature fusion algorithms and automatic acquisition of optimal dimension of feature subset.



---

*Source: 8709518-2017-07-27.xml*