# Entropy-Based Support Matrix Machine

Changming Zhu

► To cite this version:

Changming Zhu. Entropy-Based Support Matrix Machine. 2nd International Conference on Intelligence Science (ICIS), Oct 2017, Shanghai, China. pp.200-211, 10.1007/978-3-319-68121-4_21.hal-01820906

HAL Id: hal-01820906

https://inria.hal.science/hal-01820906

Submitted on 22 Jun 2018

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
Entropy-based Support Matrix Machine

Changming Zhu

College of Information Engineering, Shanghai Maritime University,
Shanghai, P.R. China 201306
cmzhu@shmtu.edu.cn

**Abstract.** Support Vector Machine (SVM) cannot process imbalanced problem and matrix patterns. Thus, Fuzzy SVM (FSVM) is proposed to process imbalanced problem while Support Matrix Machine (SMM) is proposed to process matrix patterns. FSVM applies a fuzzy membership to each training pattern such that different patterns can make different contributions to the learning machine. However, how to evaluate fuzzy membership becomes the key point to FSVM. Although SMM can process matrix patterns, it still has no ability to process imbalanced problem. This paper adopts SMM as the basic and proposes an entropy-based support matrix machine for imbalanced data sets, i.e., ESMM. The contributions of ESMM are: (1) proposing an entropy-based fuzzy membership evaluation approach which enhances importance of certainty patterns, (2) guaranteeing importance of positive patterns and getting a more flexible decision surface. Experiments on real-world imbalanced data sets and matrix patterns validate the effectiveness of ESMM.

**Keywords:** Support Matrix Machine; Entropy; Fuzzy Membership; Imbalanced Data Set; Pattern Recognition

# 1 Introduction

Support Vector Machine (SVM) constructs a hyperplane or set of hyperplanes in a high- or infinite-dimensional space, which can be used for classification, regression, or other tasks. Intuitively, a good separation is achieved by the hyperplane that has the largest distance to the nearest training-data point of any class (so-called functional margin), since in general the larger the margin the lower the generalization error of the classifier [1]. Conventional SVM can be used in many tasks including text and hypertext categorization, classification of images, classifying proteins in medical science, and recognizing hand-written characters. Although SVM has been validated effect on these applications, it still has two disadvantages. One is that SVM cannot process matrix patterns and another is that it cannot process imbalanced problem.

* Matrix patterns which dimensions are *m* × *n* (where *m* and *n* are both larger than 1) are the basic of matrix learning. For example, video and images are both matrix patterns. In order to process matrix patterns, matrix-pattern-oriented learning machine (MatC), i.e., matrix learning machine, has been developed. Classical learning machines include matrix-pattern-oriented Ho-Kashyap
learning machine with regularization learning (MatMHKS) [2], new least squares
support vector classification based on matrix patterns (MatLSSVC) [3], and one-
class support vector machines based on matrix patterns (OCSVM) [4]. Besides
those matrix learning machines, Xie et. al have proposed a Support Matrix
Machine (SMM) [5] so as to replace SVM. SMM can leverage the structure of
the data matrices and has the grouping effect property. But all of these matrix
learning machines cannot process imbalanced problem.

* As is known to all, in many real-world classification problems, such as e-mail foldering [6], fault diagnosis [7], detection of oil spills [8], and medical diagnosis [9], we can always divide a data set into two classes, one is positive class and the other is negative class. When the size of positive class is much smaller than that of negative class, imbalanced problem occurs. Since most standard classification learning machines including Support Vector Machine (SVM) and Neural Network (NN) are proposed with the assumption on the balanced class distributions or equal misclassification costs [10], so they fail to properly represent the distributive characteristics of patterns and result in the unfavorable performance when they are adopted to process imbalanced problem. In order to overcome such a disadvantage, Fuzzy SVM (FSVM) [11] and Bilateral-weighted FSVM (B-FSVM) [12] are proposed. FSVM applies a fuzzy membership to each input pattern and reformulates SVM such that different input patterns can make different contributions to the learning of decision surface. B-FSVM treats every pattern as both positive and negative classes, but with different memberships due to we can not say one pattern belongs to one class absolutely. But for both of them, how to determine the fuzzy membership function is the key point. Furthermore, both of them cannot process matrix patterns.

In this paper, we try to propose a learning machine which can process matrix
patterns and imbalanced problem. First, in order to process matrix patterns, we
adopt SMM as a basic. Then in order to process imbalanced problem, we adopt
the notion of FSVM, namely, applies a fuzzy membership to each input pattern.
Furthermore, for the fuzzy membership, we propose a new fuzzy membership
evaluation approach which assigns the fuzzy membership of each pattern based
on its class certainty. In this paper, class certainty demonstrates the certainty
of pattern labeled to a certain class. Due to the entropy is an effective measure
of certainty, we adopt the entropy to evaluate the class certainty of each pat-
tern. In doing so, the entropy-based fuzzy membership evaluation approach is
proposed. This approach determines the fuzzy membership of training patterns
based on their corresponding entropies. By adopting the entropy-based fuzzy
membership evaluation and SMM, the Entropy-based Support Matrix Machine
(ESMM) is proposed to process the imbalanced data sets. In practice, as the
importance of positive class is higher than that of negative class in imbalanced
data sets, i.e., the learning machine should pay more attention to positive pat-
terns than negative ones. Thus, positive patterns are assigned to relatively large
fuzzy memberships to guarantee the importance of positive class here. While,
the fuzzy memberships of negative patterns are determined by the entropy-based
fuzzy membership evaluation approach, i.e., patterns with lower class certainty
are assigned to small fuzzy memberships based on the criterion that patterns
with lower class certainty are more insensitive to noise, and easily mislead the
decision surface, thus their importance should be weakened on imbalanced data
sets. After evaluating the fuzzy membership of all training patterns, ESMM is
adopted to classify imbalanced data sets.

The contributions of this paper can be highlighted as follows:

(1) A new entropy-based fuzzy membership evaluation approach is proposed. This approach adopts entropy to evaluate class certainty of a pattern and determines the corresponding fuzzy membership based on class certainty. In doing so, the learning machine can pay more attention to the patterns with higher class certainty to result in more robust decision surface.

(2) To guarantee the importance of positive class, the positive patterns are assigned to the relatively large fuzzy memberships, which results in the decision surface paying more attention to the positive class so as to increase generalization of learning machine.

The rest of this paper is given below. Section II introduces the proposed
entropy-based fuzzy membership evaluation approach, then give the details of
ESMM. In Section III, several experiments on real-world imbalanced data sets
and matrix patterns including images are conducted to validate the effectiveness
of ESMM. Following that, conclusions are given in Section IV.

2 Entropy-based matrix learning machine (ESMM)

2.1 Entropy-based fuzzy membership

When we process imbalanced data sets, positive class is always more importan-
t than negative class. Thus, in the proposed entropy-based fuzzy membership
evaluation approach, we assign positive patterns to a relatively high fuzzy mem-
bership, e.g., 1.0, to guarantee the importance of positive class. As to the negative
class, we fuzzify negative patterns based on their class certainties.

In information theory, entropy is always used to characterize the certainty
of source of information. If entropy is smaller, then information is more certain
[13]. By employing this character of entropy, we can evaluate the class certain-
ty of a pattern in the training class. After getting the class certainty of each
pattern, we assign fuzzy membership of each training pattern based on class cer-
tainty. In practice, the patterns with higher class certainty, i.e., lower entropy,
are assigned to higher fuzzy memberships to enhance their contributions to the
decision surface, and vice versa.

Suppose that there are *N* training patterns, {*x*<sub>*i*</sub>, *y*<sub>*i*</sub>} where *i* = 1, 2, ..., *N* and *y*<sub>*i*</sub> ∈ {+1, -1} is the class label. When *y*<sub>*i*</sub> = 1, pattern *x*<sub>*i*</sub> belongs to the positive class, otherwise, it belongs to the negative class. The probabilities of *x*<sub>*i*</sub> belonging to positive and negative class are *p*<sub>+i</sub> and *p*<sub>-i</sub> respectively. The entropy of *x*<sub>*i*</sub> is *H*<sub>*i*</sub> = -*p*<sub>+i</sub>ln(*p*<sub>+i</sub>) - *p*<sub>-i</sub>ln(*p*<sub>-i</sub>) where *ln* represents the natural logarithm operator. Due to neighbors of a pattern can determine local information of it, thus the probability evaluation is based on its *k* nearest neighbors. For a pattern
$x_i$, we select its $k$ nearest neighbors $\{x_{i1}, x_{i2}, ..., x_{ik}\}$ at first. Then we count the number of both positive and negative class in these $k$ selected patterns and denote the numbers of patterns belonging to positive and negative class are $num_{+i}$ and $num_{-i}$ respectively. Finally, the probabilities of $x_i$ belonging to positive and negative class are calculated with $p_{+i} = \frac{num_{+i}}{k}$ and $p_{-i} = \frac{num_{-i}}{k}$. After evaluating the class probabilities of $x_i$, we can calculate its entropy.

By adopting the above entropy evaluation approach, the entropy of the negative patterns are $H = \{H_{-1}, H_{-2}, ..., H_{-n_{-}}\}$, where $n_{-}$ is the number of the negative patterns. $H_{min}$ and $H_{max}$ are the minimum and maximum entropy of $H$. The entropy-based fuzzy memberships for negative patterns are evaluated as follows.

Firstly, separate the negative patterns into $m$ subsets based on their entropy as described in Table 1, i.e., $\{Sub_k\}$ where $k = 1, 2, ..., m$.

<table><caption>**Table 1.** Algorithm of negative class separation.</caption><tr><td colspan="2">For k=1:m</td></tr><tr><td>Up=H<sub>max</sub> - <sup>k-1</sup>&frasl;<sub>m</sub> (H<sub>max</sub> - H<sub>min</sub>)</td><td></td></tr><tr><td>Low=H<sub>max</sub> - <sup>m</sup>&frasl;<sub>2</sub> (H<sub>max</sub> - H<sub>min</sub>)</td><td></td></tr><tr><td colspan="2">For i=1:n<sub>-</sub></td></tr><tr><td colspan="2">if Low &lt; H<sub>-i</sub> &le; Up</td></tr><tr><td colspan="2">negative pattern x<sub>i</sub> is distributed into the subset Sub<sub>k</sub>.</td></tr><tr><td colspan="2">End</td></tr><tr><td colspan="2">End</td></tr></table>

Then, fuzzy memberships of patterns in each subset are set as:

$$FM_k = 1.0 - \alpha \times (k - 1), \quad k = 1, 2, 3, ..., m \qquad (1)$$

where $FM_k$ is the fuzzy membership for patterns distributed in subset $Sub_k$, the fuzzy membership parameter $\alpha \in (0, \frac{1}{m-1})$ since $FM_k$ is positive and not larger than 1.0. It should be declared that patterns in the same subset are set to same fuzzy membership so that these patterns selected in the same subsets have same importance to the decision surface. Finally, the fuzzy membership $s_i$ for a training pattern $x_i$ is assigned as: if $y_i = +1$, then $s_i = 1.0$, else if $y_i = -1$ and $x_i \in Sub_k$, then $s_i = FM_k$. So far, the entropy-based fuzzy membership for the training patterns are evaluated.

## 2.2 Entropy-based support matrix machine

By adopting the evaluated entropy-based fuzzy membership which is given before, we propose the entropy-based support matrix machine (ESMM). The detailed description on ESMM is given below.

Suppose that there is a binary-class classification problem with $N$ matrix patterns $(A_i, y_i, s_i)$, $i = 1, 2, ..., N$. Here $A_i \in R^{m \times n}$ is the matrix representation of $x_i$ and its class label is $y_i \in \{+1, -1\}$. If $y_i = +1$, $x_i$ or $A_i$ belongs to class $+1$ or positive class, and then if $y_i = -1$, the pattern belongs to class $-1$ or negative class. $s_i$ is the entropy-based fuzzy membership.
The corresponding criterion function of ESMM is defined below.

$$minL(W, b) = \frac{1}{2}tr(W^T W) + \theta ||W||_{\star} + \qquad (2)$$

$$C \sum_{i=1}^{N} (s_i(1 - y_i[\mathrm{tr}(W^T A_i) + b_i]))$$

where $W \in R^{m \times n}$ is the matrix of regression coefficients and its nuclear norm is $||W||_{\star}$. $\theta$ is the coefficient. $tr$ is the trace of matrix. $b_i$ is a loose variable for pattern $A_i$. $C$ ($C \in R$, $C \ge 0$) is the regularization parameter that adjusts the trade-off between model complexity and training error. Here, $b = [b_1, ..., b_i, ..., b_N]^T$ and $b_i$ is started as $b_i \ge 0$. The iteration for $b$ is given in Eq. (3).

$$b_i(k + 1) = b_i(k) + \rho(e_i(k) + |e_i(k)|) \qquad (3)$$

where the error vector $e$ at $k$-th iteration of $A_i$, i.e., $e_i(k)$ should be $W(k)^T A_i - 1 - b_i(k)$ and $W(k)$ and $b_i(k)$ are $k$-th component of $W$ and $b_i$. Then we adopt the similar method of SMM to get the optimal $W$ and $b$ here. After that, we can get the discriminant function of ESMM as below.

$$g(A_i) = W^T A_i \qquad (4)$$

If $g(A_i) > 0$, then we label $A_i$ as a positive pattern, then if $g(A_i) < 0$, we label $A_i$ as a negative pattern.

# 3 Experiments

In this section, we adopt 25 real-world imbalanced data sets and 5 image data sets for examples and the compared learning machines are SVM, FSVM, MatMHKS, B-FSVM, SMM. The used 25 real-world imbalanced data sets are selected from the KEEL imbalanced benchmark ones [14, 15]. Information of these data sets are given in Tables 2 and 3.

## 3.1 Experimental settings

In terms of the compared learning machines and ESMM, the experimental settings are given here. For the SVM-based learning machines, the used kernel is Radial Basis Function (RBF) kernel $ker(x_i, x_j) = \exp(-\frac{||x_i - x_j||_2^2}{\sigma^2})$ where $\sigma$ is selected from the set $\{10^{-3}, 10^{-2}, ..., 100, 1000\}$. For MatMHKS, the experimental setting can be found in [2]. For SMM, the experimental setting can be found in [5]. For ESMM, its setting is similar with SMM. Moreover, for ESMM, the number of the separated subsets $m = 10$ and the fuzzy membership parameter $\alpha = 0.05$ which results in the fuzzy membership $0.5 \le s_i \le 1.0$. The reason of restricting $s_i \in [0.5, 1.0]$ is that the class label of $x_i$ should not be neglected when determining the fuzzy membership, i.e., $x_i$ is more likely to be classified to the
<table><caption>**Table 2.** Information of real-world imbalanced data sets (IR represents the imbalanced ratio of the corresponding data set).</caption><thead><tr><th>Order</th><th>Data set</th><th>Attributes</th><th>Training</th><th>Testing</th><th>IR</th></tr></thead><tbody><tr><td>1</td><td>Wisconsin</td><td>9</td><td>546</td><td>137</td><td>1.86</td></tr><tr><td>2</td><td>Haberman</td><td>3</td><td>245</td><td>61</td><td>2.78</td></tr><tr><td>3</td><td>Newthyroid2</td><td>5</td><td>172</td><td>43</td><td>5.14</td></tr><tr><td>4</td><td>Ecoli3</td><td>7</td><td>269</td><td>67</td><td>8.6</td></tr><tr><td>5</td><td>Ecoli046vs5</td><td>6</td><td>162</td><td>41</td><td>9.15</td></tr><tr><td>6</td><td>Yeast2vs4</td><td>8</td><td>411</td><td>103</td><td>9.08</td></tr><tr><td>7</td><td>Ecoli067vs5</td><td>6</td><td>176</td><td>44</td><td>10</td></tr><tr><td>8</td><td>Led7digit02456789vs1</td><td>7</td><td>354</td><td>89</td><td>10.97</td></tr><tr><td>9</td><td>Glass2</td><td>9</td><td>171</td><td>43</td><td>11.59</td></tr><tr><td>10</td><td>Yeast1vs7</td><td>7</td><td>367</td><td>92</td><td>14.3</td></tr><tr><td>11</td><td>Pageblocks13vs4</td><td>10</td><td>378</td><td>94</td><td>15.86</td></tr><tr><td>12</td><td>Glass016vs5</td><td>9</td><td>147</td><td>37</td><td>19.44</td></tr><tr><td>13</td><td>Yeast2vs8</td><td>8</td><td>386</td><td>96</td><td>23.1</td></tr><tr><td>14</td><td>Yeast1289vs7</td><td>8</td><td>758</td><td>189</td><td>30.57</td></tr><tr><td>15</td><td>Yeast6</td><td>8</td><td>1187</td><td>297</td><td>41.4</td></tr><tr><td>16</td><td>Pima</td><td>8</td><td>614</td><td>154</td><td>1.87</td></tr><tr><td>17</td><td>Ecoli1</td><td>7</td><td>269</td><td>67</td><td>3.36</td></tr><tr><td>18</td><td>Segment0</td><td>19</td><td>1846</td><td>462</td><td>6.02</td></tr><tr><td>19</td><td>Ecoli034vs5</td><td>7</td><td>160</td><td>40</td><td>9</td></tr><tr><td>20</td><td>Ecoli01vs235</td><td>7</td><td>195</td><td>49</td><td>9.17</td></tr><tr><td>21</td><td>Yeast05679vs4</td><td>8</td><td>422</td><td>106</td><td>9.35</td></tr><tr><td>22</td><td>Glass016vs2</td><td>9</td><td>154</td><td>38</td><td>10.29</td></tr><tr><td>23</td><td>Glass0146vs2</td><td>9</td><td>164</td><td>41</td><td>11.06</td></tr><tr><td>24</td><td>Cleveland0vs4</td><td>13</td><td>138</td><td>35</td><td>12.31</td></tr><tr><td>25</td><td>Ecoli4</td><td>7</td><td>269</td><td>67</td><td>15.8</td></tr></tbody></table>

<table><caption>**Table 3.** Information of image data sets. Imbalanced ratio of each image data set is 9.0.</caption><thead><tr><th>Order</th><th>Data set</th><th>Attributes</th><th>Training</th><th>Testing</th></tr></thead><tbody><tr><td>26</td><td>COIL-20 [16]</td><td>32x32</td><td>1296</td><td>144</td></tr><tr><td>27</td><td>Letter [17]</td><td>24x18</td><td>450</td><td>50</td></tr><tr><td>28</td><td>ORL [18]</td><td>32x20</td><td>360</td><td>40</td></tr><tr><td>29</td><td>PubFig [19]</td><td>74x74</td><td>52917</td><td>5880</td></tr><tr><td>30</td><td>Gaze [20]</td><td>5184x3456</td><td>5292</td><td>588</td></tr></tbody></table>

class which is indicated by its class label. As to negative class patterns, we set $s_i > 0.5$ to indicate that a negative pattern $x_i$ is more likely to belong to the negative class. Moreover, the number of nearest neighbors $k$ for calculating the class probability is selected from $\{1, 2, 3, ..., 18, 19, 20\}$. In order to measure the performance of compared learning machines on imbalanced data sets, the values of Area Under the ROC Curve (AUC) [25] is given. Besides those, $maxIter = 500$ represents the maximal size of the iteration. One-against-one classification strategy is used for multi-class problems here [21–24]. The 10-fold cross validation approach [26] is adopted for the parameter selection. The computations are performed on Intel Core 2 processors with 2.66GHz, 8G RAM, Microsoft Windows 7, and Matlab environment.

## 3.2 Experiments on real-world imbalanced data sets

For experiments, the AUC values are presented in Table 4. The average AUC on all used data sets are presented. Moreover, the average ranks of the learning machines on the used data sets are listed.

From the experimental results, it is found that: (1) ESMM results in the best classification performance on 12 of 30 imbalanced data sets which indicates that our proposal outperforms the compared learning machines; (2) ESMM outperforms the traditional MatMHKS on 26 of 30 imbalanced data sets; (3) The average AUC of ESMM is respectively greater than B-FSVM, FSVM, and SVM
<table><caption>**Table 4.** AUC values (%) of the compared learning machines on real-world imbalanced data sets and image data sets. The best *k* for each data sets of ESMM is presented. (Note that the average AUC values and the average ranks of the compared learning machines are listed in the last two rows. Best result for each data set is in bold.)</caption><thead><tr><th>Data set</th><th>k</th><th>ESMM</th><th>SMM</th><th>MatMHKS</th><th>B-FSVM</th><th>FSVM</th><th>SVM</th></tr></thead><tbody><tr><td>1</td><td>9</td><td>94.05 ± 2.37</td><td>94.02 ± 2.30</td><td>89.98 ± 3.33</td><td>91.30 ± 3.03</td><td>96.10 ± 2.41</td><td>93.64 ± 3.67</td></tr><tr><td>2</td><td>13</td><td>64.42 ± 1.96</td><td>64.47 ± 1.98</td><td>67.06 ± 1.40</td><td>61.58 ± 1.44</td><td>58.00 ± 2.05</td><td>54.08 ± 1.65</td></tr><tr><td>3</td><td>11</td><td>93.28 ± 1.87</td><td>91.47 ± 1.89</td><td>89.78 ± 2.57</td><td>92.44 ± 2.32</td><td>93.43 ± 2.33</td><td>97.12 ± 2.90</td></tr><tr><td>4</td><td>17</td><td>87.66 ± 2.41</td><td>87.37 ± 2.41</td><td>93.67 ± 2.70</td><td>94.55 ± 2.54</td><td>76.83 ± 2.36</td><td>65.74 ± 1.80</td></tr><tr><td>5</td><td>14</td><td>98.61 ± 2.73</td><td>97.15 ± 2.74</td><td>92.59 ± 2.91</td><td>89.07 ± 3.15</td><td>86.61 ± 1.06</td><td>82.43 ± 1.91</td></tr><tr><td>6</td><td>7</td><td>93.84 ± 2.95</td><td>94.33 ± 2.94</td><td>91.79 ± 1.47</td><td>87.37 ± 2.11</td><td>87.32 ± 1.85</td><td>83.77 ± 2.14</td></tr><tr><td>7</td><td>5</td><td>97.12 ± 2.94</td><td>98.67 ± 2.87</td><td>90.79 ± 1.71</td><td>94.39 ± 1.97</td><td>78.90 ± 2.28</td><td>74.15 ± 2.52</td></tr><tr><td>8</td><td>17</td><td>92.80 ± 2.62</td><td>90.53 ± 2.58</td><td>88.30 ± 2.29</td><td>93.43 ± 2.35</td><td>87.66 ± 2.14</td><td>81.64 ± 2.87</td></tr><tr><td>9</td><td>18</td><td>81.33 ± 2.22</td><td>80.07 ± 2.20</td><td>68.12 ± 1.66</td><td>66.99 ± 1.74</td><td>69.21 ± 2.09</td><td>51.76 ± 1.51</td></tr><tr><td>10</td><td>18</td><td>78.41 ± 2.53</td><td>76.58 ± 2.50</td><td>58.24 ± 1.95</td><td>68.95 ± 1.85</td><td>61.56 ± 2.14</td><td>67.10 ± 2.20</td></tr><tr><td>11</td><td>15</td><td>79.86 ± 2.36</td><td>79.29 ± 2.29</td><td>77.65 ± 1.93</td><td>80.99 ± 2.37</td><td>92.11 ± 2.67</td><td>91.22 ± 3.32</td></tr><tr><td>12</td><td>14</td><td>87.24 ± 2.27</td><td>85.75 ± 2.21</td><td>73.53 ± 2.07</td><td>80.66 ± 1.91</td><td>74.85 ± 2.18</td><td>75.31 ± 1.74</td></tr><tr><td>13</td><td>15</td><td>71.30 ± 1.67</td><td>69.28 ± 1.69</td><td>67.12 ± 1.55</td><td>56.86 ± 1.20</td><td>57.62 ± 1.87</td><td>62.45 ± 1.52</td></tr><tr><td>14</td><td>10</td><td>91.00 ± 2.24</td><td>91.62 ± 2.22</td><td>92.81 ± 3.17</td><td>87.97 ± 2.80</td><td>71.63 ± 1.54</td><td>71.64 ± 2.13</td></tr><tr><td>15</td><td>5</td><td>72.58 ± 1.31</td><td>72.84 ± 1.32</td><td>71.15 ± 2.13</td><td>69.81 ± 2.38</td><td>68.65 ± 1.53</td><td>71.50 ± 1.75</td></tr><tr><td>16</td><td>3</td><td>90.36 ± 1.72</td><td>89.80 ± 1.72</td><td>81.80 ± 2.18</td><td>93.74 ± 2.44</td><td>85.40 ± 2.59</td><td>78.11 ± 2.36</td></tr><tr><td>17</td><td>10</td><td>80.62 ± 2.19</td><td>78.88 ± 2.20</td><td>82.32 ± 2.82</td><td>83.40 ± 2.01</td><td>98.24 ± 2.03</td><td>99.72 ± 2.25</td></tr><tr><td>18</td><td>6</td><td>93.52 ± 3.00</td><td>91.50 ± 2.89</td><td>87.46 ± 1.91</td><td>83.81 ± 1.88</td><td>89.61 ± 2.98</td><td>77.26 ± 2.20</td></tr><tr><td>19</td><td>18</td><td>92.35 ± 2.69</td><td>93.72 ± 2.73</td><td>92.22 ± 3.10</td><td>91.20 ± 2.19</td><td>74.69 ± 2.30</td><td>82.43 ± 2.18</td></tr><tr><td>20</td><td>19</td><td>83.26 ± 1.96</td><td>81.00 ± 1.99</td><td>78.59 ± 2.28</td><td>79.42 ± 2.16</td><td>64.48 ± 2.02</td><td>66.34 ± 1.74</td></tr><tr><td>21</td><td>5</td><td>75.38 ± 2.08</td><td>75.26 ± 2.06</td><td>69.36 ± 2.09</td><td>55.51 ± 1.57</td><td>61.42 ± 1.84</td><td>53.73 ± 1.52</td></tr><tr><td>22</td><td>16</td><td>91.84 ± 2.19</td><td>91.08 ± 2.18</td><td>85.21 ± 2.01</td><td>95.55 ± 3.40</td><td>87.28 ± 2.53</td><td>94.66 ± 2.85</td></tr><tr><td>23</td><td>4</td><td>81.49 ± 2.00</td><td>80.37 ± 2.00</td><td>59.32 ± 1.60</td><td>71.62 ± 2.15</td><td>66.51 ± 2.05</td><td>55.52 ± 1.35</td></tr><tr><td>24</td><td>1</td><td>74.72 ± 2.07</td><td>74.44 ± 2.01</td><td>63.24 ± 1.72</td><td>60.41 ± 1.21</td><td>68.05 ± 1.76</td><td>77.90 ± 1.86</td></tr><tr><td>25</td><td>12</td><td>95.59 ± 2.79</td><td>96.65 ± 2.81</td><td>95.98 ± 2.51</td><td>92.53 ± 2.80</td><td>84.58 ± 1.72</td><td>77.94 ± 2.37</td></tr><tr><td>26</td><td>5</td><td>65.40 ± 1.78</td><td>64.90 ± 1.74</td><td>59.33 ± 1.97</td><td>71.77 ± 1.76</td><td>71.07 ± 1.76</td><td>69.50 ± 2.37</td></tr><tr><td>27</td><td>18</td><td>66.08 ± 2.15</td><td>65.88 ± 2.15</td><td>57.58 ± 1.01</td><td>60.62 ± 1.50</td><td>49.37 ± 1.45</td><td>48.94 ± 1.32</td></tr><tr><td>28</td><td>20</td><td>90.19 ± 2.06</td><td>88.54 ± 2.01</td><td>86.21 ± 3.07</td><td>79.27 ± 2.02</td><td>79.82 ± 1.71</td><td>82.30 ± 2.10</td></tr><tr><td>29</td><td>8</td><td>92.89 ± 2.01</td><td>93.59 ± 1.99</td><td>88.12 ± 2.36</td><td>95.14 ± 3.19</td><td>89.48 ± 2.26</td><td>81.30 ± 2.45</td></tr><tr><td>30</td><td>18</td><td>94.78 ± 1.70</td><td>92.15 ± 1.68</td><td>90.61 ± 2.56</td><td>87.60 ± 2.02</td><td>66.13 ± 2.23</td><td>55.69 ± 2.21</td></tr><tr><td>AVG.</td><td>11.72</td><td>85.06 ± 2.23</td><td>84.37 ± 2.21</td><td>79.66 ± 2.20</td><td>80.60 ± 2.18</td><td>76.55 ± 2.02</td><td>75.16 ± 2.16</td></tr><tr><td>AVG.rank</td><td></td><td>2.07</td><td>2.53</td><td>4.03</td><td>3.50</td><td>4.30</td><td>4.57</td></tr></tbody></table>

on about 5%, 9%, and 10%, which demonstrates that ESMM is of significant advantage in processing imbalanced data sets compared to SVM, FSVM, and B-FSVM; (4) It is found that for some data sets, for example, COIL-20, the three matrix-pattern-oriented approaches perform worse than SVM. We think it is the coincidence due to the training part is gotten in random. But according to the average result, we still find that our proposed ESMM outperforms others in average; (5) For the used 25 vector data sets, ESMM performs best on the 9 data sets of them. For others, ESMM performs better in average and it does not perform worst on any vector data set. This phenomenon can explain the superiority of the ESMM on the vector data sets; (6) As we said before, ESMM and FSVM both adopt fuzzy membership to each input pattern. Now from the experiments, it is found that compared with FSVM, the better performance of ESMM validates that the proposed entropy-based fuzzy membership evaluation approach boosts the performance of a learning machine.

## 3.3 Influence of parameter *k* on the performance of ESMM

In ESMM, the entropy-based fuzzy membership is evaluated based on the class probability of each training pattern. Thus, the number of nearest neighbors *k* might have some influence on the class probability. To further investigate the effectiveness of ESMM, here, we study the influence of *k* on the classification performance. All data sets given in Table 2 are used and related experimental results are given in Fig. 1. The figure shows AUC on the testing sets of the
adopted real-world imbalanced data sets and image data sets with respect to *k*. It is found that: (1) the number of the selected nearest neighbors for calculating the class probability, i.e., *k*, has some influence on the classification performance since AUC curves fluctuate with respect to *k* on most data sets; (2) on some data sets, the classification performances are sensitive to the variation of *k*, since AUC curves on these data sets fluctuate greatly while on some data sets, are not; (3) in generally, with *k* from 8 to 12, ESMM always gets best performance. Such a result can give us a guidance that how to determine an appropriate *k* in practical use.

## 3.4 Comparison between ESMM and entropy-based MatMHKS (EMatMHKS)

Here, we give the comparison between ESMM and our previous proposed learning machine, entropy-based MatMHKS (EMatMHKS) [28] which is also an entropy-based matrix learning machine. Table 5 shows the comparison between ESMM and EMatMHKS. ↑ (↓) represents that ESMM performance better (worse) than EMatMHKS and (★) represents that the AUC value of ESMM is ★ larger or smaller than the one of EMatMHKS. From this table, we find that ESMM performs better than EMatMHKS in average.

<table><caption>Table 5. Comparison between ESMM and EMatMHKS on AUC values (%).</caption><thead><tr><th>Data set</th><th>Data set</th><th>Data set</th><th>Data set</th><th>Data set</th><th>Data set</th><th>Data set</th></tr></thead><tbody><tr><td>1</td><td>↑(1.02)</td><td>2</td><td>↑(0.03)</td><td>3</td><td>↑(2.11)</td><td>4</td><td>↑(0.02)</td><td>5</td><td>↑(1.01)</td><td>6</td><td>↑(0.78)</td></tr><tr><td>7</td><td>↓(0.23)</td><td>8</td><td>↑(1.02)</td><td>9</td><td>↑(2.98)</td><td>10</td><td>↑(6.18)</td><td>11</td><td>↓(0.03)</td><td>12</td><td>↑(3.91)</td></tr><tr><td>13</td><td>↑(0.82)</td><td>14</td><td>↑(0.56)</td><td>15</td><td>↑(0.81)</td><td>16</td><td>↑(2.87)</td><td>17</td><td>↓(0.12)</td><td>18</td><td>↑(1.98)</td></tr><tr><td>19</td><td>↑(0.04)</td><td>20</td><td>↑(1.23)</td><td>21</td><td>↑(3.12)</td><td>22</td><td>↑(2.91)</td><td>23</td><td>↑(8.19)</td><td>24</td><td>↑(2.12)</td></tr><tr><td>25</td><td>↑(0.01)</td><td>26</td><td>↑(2.01)</td><td>27</td><td>↑(2.10)</td><td>28</td><td>↑(1.81)</td><td>29</td><td>↑(1.07)</td><td>30</td><td>↑(0.91)</td></tr></tbody></table>

# 4 Conclusion

There are two hot spots of present research, one is imbalanced problem and the other is matrix learning. Imbalanced problem occurs when the size of positive class is more smaller than that of negative class. However, most standard classification learning machines result in unfavorable performance on imbalanced data sets since they are originally designed for processing balanced problems. Although SVM can process imbalanced data sets in some extent, it assigns the same importance to each training pattern. This results in the decision surfaces biasing toward the negative class. In order to overcome the disadvantage of SVM, some researchers propose FSVM and B-FSVM by applying fuzzy memberships to the training patterns to reflect different importance of them. Since the key point in FSVM and B-FSVM is how to determine the fuzzy membership, so this paper presents an entropy-based fuzzy membership evaluation approach for imbalanced data sets. Moreover, matrix patterns cannot be solved by those traditional learning machines including SVM well, so some scholars have developed
<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Variation of AUC values of ESMM with respect to *k* on the adopted imbalanced data sets and image data sets. Here, in the legend, each number denotes one data set which is given in Table 2 or Table 3.</figcaption></figure>

MatMHKS and SMM. This paper adopts the entropy-based fuzzy membership and SMM, and then proposes ESMM. ESMM can not only guarantee importance of the positive class, but also pay more attentions to the patterns with higher class certainties. Thus, ESMM can results in more flexible decision surfaces than
both conventional SVM, FSVM, B-FSVM, and MatMHKS on the imbalanced data sets. To validate the effectiveness of ESMM, we adopt 25 real-world imbalanced data sets and 5 image data sets for experiments. Experimental results demonstrates that ESMM outperforms the compared learning machines on real-world imbalanced data sets and the images. Moreover, in the process of ESMM, *k* has some influence on the classification performance.

## Acknowledgment

This work is supported by Natural Science Foundation of Shanghai under grant number 16ZR1414500 and National Natural Science Foundation of China under grant number 61602296 and 51575336, and the authors would like to thank their supports.

## References

1. Support Vector Machine, https://en.wikipedia.org/wiki/Support_vector_machine#Applications

2. Chen, S.C., Wang, Z., Tian, Y.J.: Matrix-pattern-oriented Ho-Kashyap classifier with regularization learning. Pattern Recognition. 40(5), 533-543 (2007)

3. Wang, Z., Chen, S.C.: New least squares support vector machines based on matrix patterns. Neural Processing Letters. 26(1), 41-56 (2007)

4. Yan, Y., Wang, Q., Ni, G., Pan, Z., Kong, R.: One-class support vector machines based on matrix patterns. International Conference on Informatics, Cybernetics, and Computer Engineering. pp. 223-231 (2012)

5. Xie, Y.B., Zhang, Z.H., Li, W.J.: Support Matrix Machines. The 32nd International Conference on Machine Learning (ICML 2015).

6. Dai, H.: Class Imbalance Learning via a Fuzzy Total Margin based Support Vector Machine. Applied Soft Computing. 31, 172-184 (2015)

7. Deng, X., Tian, X.: Nonlinear Process Fault Pattern Recognition Using Statistics Kernel PCA Similarity Factor. Neurocomputing. 121, 298-308 (2013)

8. Guo, Y., Zhang, H.Z.: Oil Spill Detection Using Synthetic Aperture Radar Images and Feature Selection in Shape Space. International Journal of Applied Earth Observation and Geoinformation. 30, 146-157 (2014)

9. Ozcift, A., Gulten, A.: Classifier Ensemble Construction with Rotation Forest to Improve Medical Diagnosis Performance of Machine Learning Algorithms. Computer Methods and Programs in Biomedicine. 104(3), 443-451 (2011)

10. Brown, I., Mues, C.: An Experimental Comparison of Classification Algorithms for Imbalanced Credit Scoring Data Sets. Expert Systems with Applications. 39(3), 3446-3453 (2012)

11. Lin, C. Wang, S.: Fuzzy Support Vector Machines. IEEE Transactions on Neural Networks. 13(2), 464-471 (2002)

12. Wang, Y., Wang, S., Lai, K.K.: A New Fuzzy Support Vector Machine to Evaluate Credit Risk. IEEE Transactions on Fuzzy Systems. 13(6), 820-831 (2005)

13. Shannon, C.E.: A Mathematical Theory of Communication. SIGMOBILE Mobile Computing and Communications Review. 5(1), 3-55 (2001)

14. Alcala-Fdez, J., Fernandez, A., Luengo, J., Derrac, J., Garcia, S., Sanchez, L., and Herrera, F.: A Software Tool to Assess Evolutionary Algorithms for Data Mining Problems. Journal of Multiple-Valued Logic and Soft Computing. 17, 2-3 (2011)
15. Alcala-Fdez, J., Sanchez, L., Garcia, S., Jesus, M.J., Ventura, S., Garrell, J.M., Otero, J., Romero, C., Bacardit, J., Rivas, V.M.: A Software Tool to Assess Evolutionary Algorithms for Data Mining Problems. Soft Computing. 13(3), 307–318 (2009)

16. Nene, S.A., Nayar, S.K., Murase, H.: Columbia Object Image Library (COIL-20). Technical Report CUCS-005-96, Columbia University (1996)

17. Cun, B.B.L., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., Jackel, L.D.: Handwritten digit recognition with a back-propagation network. Advances in neural information processing systems (1990)

18. Bennett, F., Richardson, T., Harter, A.: Teleporting-making applications mobile. Mobile Computing Systems and Applications. pp. 82–84 (1994)

19. Kumar, N., Berg, A.C., Belhumeur, P.N., Nayar, S.K.: Attribute and simile classifiers for face verification. International Conference on Computer Vision. pp. 365–372 (2009)

20. Smith, B.A., Yin, Q., Feiner, S.K., Nayar, S.K.: Gaze Locking, Passive Eye Contact Detection for Human-Object Interaction. ACM Symposium on User Interface Software and Technology (UIST). pp. 271–280 (2013)

21. Milgram, J., Cheriet, M., Sabourin, R.: "One Against One" or "One Against All", Which One is Better for Handwriting Recognition with SVMs?. Tenth International Workshop on Frontiers in Handwriting Recognition. (2013)

22. Debnath, R., Takahide, N., Takahashi, H.: A decision based one-against-one method for multi-class support vector machine. Pattern analysis and applications. 7(2), 164–175 (2004)

23. Hsu, C.W., Lin, C.J.: A comparison of methods for multiclass support vector machines. IEEE Transactions on Neural Networks. 13(2), 415–425 (2002)

24. Cortes, C., Vapnik, V.: Support vector machine, Machine Learning. 20(3), 273–297 (1995)

25. Huang, J., Ling, C.X.: Using AUC and Accuracy in Evaluating Learning Algorithms. IEEE Transactions on Knowledge and Data Engineering. 17(3), 299–310 (2005)

26. Braga-Neto, U.M., Dougherty, E.R.: Is Cross-validation Valid for Small-sample Microarray Classification? Bioinformatics. 20(3), 374–380 (2004)

27. Demsar, J.: Statistical comparisons of classifiers over multiple data sets. The Journal of Machine Learning Research. 7, 1–30 (2006)

28. Zhu, C.M., Wang, Z.: Entropy-based matrix learning machine for imbalanced data sets. Pattern Recognition Letters. 88(1), 72–80 (2017)