# A New Method for Solving Supervised Data Classification Problems

Authors: Parvaneh Shabanzadeh; Rubiyah Yusof
Journal: Abstract and Applied Analysis
 (2014)

---

## Abstract

Supervised data classification is one of the techniques used to extract nontrivial information from data. Classification is a widely used technique in various fields, including data mining, industry, medicine, science, and law. This paper considers a new algorithm for supervised data classification problems associated with the cluster analysis. The mathematical formulations for this algorithm are based on nonsmooth, nonconvex optimization. A new algorithm for solving this optimization problem is utilized. The new algorithm uses a derivative-free technique, with robustness and efficiency. To improve classification performance and efficiency in generating classification model, a new feature selection algorithm based on techniques of convex programming is suggested. Proposed methods are tested on real-world datasets. Results of numerical experiments have been presented which demonstrate the effectiveness of the proposed algorithms.


---

## Body

## 1. Introduction

Supervised data classification is a widely used technique in various fields, including data mining, whose aim is to establish rules for the classification of some observations assuming that the classes of data are known. Due to the explosive growth of both business and scientific databases, extracting efficient classification rules from such databases is of major importance.In the past 30 decades various algorithms were designed for supervised data classification which are based on completely different approaches, for example, statistics methods [1], neural networks [2], genetic algorithms [3], graphical models [4], and adaptive spline methods [5].Algorithms based on inductive logic programming [6] and hybrid systems [7] are also used for supervised data classification. Kotsiantis in 2007 and Mangasarian and Musicant in 2001 [8, 9] presented good review of these approaches, including their definition and comparison. One of the new and most promising approaches to supervised data classification is based on methods of mathematical optimization. There exist different ways for the application of optimization; for example, see [10–12]. One of these methods is based on finding clusters for the given training sets. The data vectors are allocated to the closest cluster and correspondingly to the set, which contains this cluster [13].On the other hand, one of the most influencing important factors on the classification accuracy rate is feature selection. If the dataset contains a number of features, the dimension of the space will be large and nonclean, degrading the classification accuracy rate [14]. An efficient and robust feature selection method can eliminate noisy, irrelevant, and redundant data [15]. Therefore reducing it without loss of useful information is expected to accelerate the algorithms and increase the accuracy. Most feature selections are based on statistical considerations, and the features are usually removed according to a correlation between observations and features (see [15, 16]). In [17, 18] approaches based on optimization techniques have been developed. Then a new feature selection algorithm based on techniques of convex programming is proposed.So in this research, new algorithms for classification and feature selection problems based on optimization techniques are designed; for the execution of these approaches one needs to solve complex problems of nonconvex and nonsmooth unconstrained optimization, either local or global. Despite the nonsmoothness and nonconvexity of the objective functions, global methods are much simpler and more applicable than local ones. In the present research, we are adapted and used as one type of direct global optimization methods, namely, mesh adaptive direct search (MADS) [19]. MADS is derivative-free method in the sense that this method does not compute nor even attempt to evaluate derivatives. Mesh adaptive direct search methods are designed to only use function values and require only a numerical value of the objective; no knowledge about the internal structure of the problem is needed [19].Results of computational experiments using real-world datasets were presented and compared with the best known solutions from the literature.The paper is organized as follows. The optimization approach to classification is considered in Section2. In Section 3, an algorithm for feature selection problems is studied. In Section 4 an algorithm is presented for solving optimization problems. The discussion of the results of computational experiments and their analysis is explained in Section 5. Finally Section 6 concludes the paper.


## 2. A New Optimization Algorithm for Solving Classification Problem

Consider a setD consisting of m points d
1
,
…
,
d
m which contains k classes, that is, k nonempty finite subsets D
j
,
j
=
1
,
…
,
k of n-dimensional space R
n. Assume that the set D
j consists of e
j points (
j
=
1
,
…
,
k
). The aim of classification is to categorize a new observation into one of the known classes and there are many existing approaches for solving this problem (as mentioned in Introduction). In continue, a classification method that is based on optimization ways has been studied. Numerical experiments verify that this method outperforms known ones for real-world databases. In order to solve this problem, the clusters of each class of dataset have to be identified, which must be done along with centers of the corresponding clusters. New observations are allocated to the class with least distance between its centers.Thus, at the first finding the clusters of a finite set will be explained. Clustering inn-dimensional Euclidean space R
n is based on some similarity (distance) metric, the Minkowski metric was used for this aim. There are various methods for solving clustering problem. One of the most popular methods is the center based clustering model [20–22].Consider the setD; suppose that this set consists of only one cluster; thus its center can be calculated by solving the following convex programming problem:
(1)
Min
⁡
f
1
y
=
∑
i
=
1
m
y
-
d
i
s
.
t
.
y
∈
R
n
.
Suppose that y
1 is the solution of problem (1); in order to find a center of the second cluster, find the answer of the following optimization problem:
(2)
Min
f
2
y
=
∑
i
=
1
m
min
⁡
y
1
-
d
i
,
y
-
d
i
s
.
t
.
y
∈
R
n
.In the same manner, suppose the already calculated(
h
-
1
) centers, and then the center y
h of hth cluster is described as a solution to the following problem:
(3)
Min
f
h
y
=
∑
i
=
1
m
min
⁡
y
1
-
d
i
,
y
2
-
d
i
,
…
,
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
y
h
-
1
-
d
i
,
y
-
d
i
s
.
t
.
y
∈
R
n
.Then the following algorithm for solving a classification problem is proposed. Suppose that database contains 2 classes:D
1 and D
2. Let M
1
=
1
,
…
,
D
1, M
2
=
1
,
…
,
D
2, and ϵ
>
0 a tolerance.Algorithm 1.
A new algorithm for classification problem is presented.
Step
1 (initialization). Suppose that sets B
1 and B
2 contain a unique cluster; calculate the centers of clusters by solving the following problems:
(4)
Min
⁡
∑
i
∈
M
1
y
1
-
d
i
Min
⁡
∑
i
∈
M
2
y
2
-
d
i
.
Suppose thaty
11
*, y
21
* are the solutions to these problems and allow f
11
* and f
21
* to be the values of these problems, respectively. Let h
=
1.
Step
2 (identify the sets of points “misclassified” by the current clusters). Compute the sets
(5)
M
1
h
*
=
i
∈
M
1
:
min
⁡
r
=
1
,
…
,
h
⁡
y
2
r
*
-
d
i
≤
min
⁡
r
=
1
,
…
,
h
⁡
y
1
r
*
-
d
i
M
2
h
*
=
i
∈
M
2
:
min
⁡
r
=
1
,
…
,
h
⁡
y
1
r
*
-
d
i
≤
min
⁡
r
=
1
,
…
,
h
⁡
y
2
r
*
-
d
i
.
Step
3. If h
≠
1, compute the following sets:
(6)
S
1
=
i
∈
M
1
∖
M
1
h
*
:
y
1
h
*
-
d
i
≤
min
⁡
r
=
1
,
…
,
h
-
1
⁡
y
1
r
*
-
d
i
S
2
=
i
∈
M
2
∖
M
2
h
*
:
y
2
h
*
-
d
i
≤
min
⁡
r
=
1
,
…
,
h
-
1
⁡
y
2
r
*
-
d
i
.
Else
(7)
S
1
=
i
∈
M
1
∖
M
1
h
*
S
2
=
i
∈
M
2
∖
M
2
h
*
.
Step
4. Improve the center of the cluster by solving the following convex programming problems:
(8)
Min
∑
i
∈
S
1
y
1
-
d
i
(9)
Min
⁡
∑
i
∈
S
2
y
2
-
d
is.t. y
j
∈
R
n
,
j
=
1,2.
Allowy
01 and y
02 to be the solutions of the problems (8) and (9), respectively. Set y
1
h
*
=
y
01 andy
2
h
*
=
y
02.
Step
5 (checking the stopping criterion). Ifh
≠
1, calculate these functions:
(10)
f
1
h
=
∑
i
∈
M
1
min
⁡
y
1
h
*
-
d
i
,
y
11
*
-
d
i
,
…
,
y
1
,
h
-
1
*
-
d
i
f
2
h
=
∑
i
∈
M
2
min
⁡
y
2
h
*
-
d
i
,
y
21
*
-
d
i
,
…
,
y
2
,
h
-
1
*
-
d
i
and if f
1
h
-
f
1
,
h
-
1
/
f
11
,
f
2
h
-
f
2
,
h
-
1
/
f
21
<
ε, then the algorithm ends. Otherwise go to Step 6.
Step
6 (determine the estimate of next cluster). Solve the following optimization problems:
(11)
Min
⁡
∑
i
∈
M
1
min
⁡
y
1
-
d
i
,
y
11
*
-
d
i
,
…
,
y
1
h
*
-
d
i
(12)
Min
⁡
∑
i
∈
M
2
min
⁡
y
2
-
d
i
,
y
21
*
-
d
i
,
…
,
y
2
h
*
-
d
is.t. y
j
∈
R
n
,
j
=
1,2.
Step
7. Allow y
11and y
21 to be the solutions of the problems (11) and (12), respectively. Set y
1
,
h
+
1
*
=
y
11, y
2
,
h
+
1
*
=
y
21, and h
=
h
+
1 and go to Step 2.


## 3. Feature Selection Algorithm

Feature selection is concerned with the identification of a subset of features that significantly contributes to the discrimination or prediction problem. The main goal of feature selection is to search for an optimal feature subset from the initial feature set that leads to improved classification performance and efficiency in generating classification model. During the past decades, wide research has been conducted by researchers from multidisciplinary fields including data mining, pattern recognition, statistics, and machine learning. In [23] a comparison of various feature selection algorithms for large datasets is presented.Consider a database which contains 2 nonempty finite setsB
j
⊂
R
m
,
j
=
1,2. Let M
j
=
1
,
…
,
B
j
,
j
=
1,2, where B denotes the cardinality of a finite set B. Let T
j
ϵ
1,2
,
…
,
j
=
1,2
,
3 be the thresholds and let ε
>
0 be some tolerance.Algorithm 2.
Feature selection.
Step
1 (initialization). Sett
=
1
,
I
t
=
1
,
…
,
m.
Step
2. Find centers of clusters by assuming that the sets B
j
,
j
=
1,2 contain a unique cluster. Compute the centers of clusters by solving the following problems of convex programming:
(13)
Minimize
∑
i
∈
M
j
x
j
-
b
i
j
q
subject
to
x
j
ϵ
R
m
,
j
=
1,2
.
Here x
q is defined by x
q
=
(
∑
t
=
1
m
x
t
q
)
1
/
q.
Step
3. Find points of the set B
j
,
j
=
1,2 which are closer to the cluster center of the other set (bad points).
Letx
*
j
,
j
=
1,2 be solutions to (13). Compute the sets
(14)
M
1
t
=
i
ϵ
M
j
:
x
*
2
-
b
i
1
q
≤
x
*
1
-
b
i
1
q
M
2
t
=
i
ϵ
M
2
:
x
*
1
-
b
i
2
q
≤
x
*
2
-
b
i
2
q
.
Set M
3
t
=
M
1
t
∪
M
2
t.
Ift
=
0, then go to Step 5; otherwise go to Step 4.
Step
4. Calculate
(15)
E
N
i
=
M
i
t
-
M
i
t
-
1
,
i
=
1,2
,
3
f
f
i
=
E
N
i
-
T
i
,
i
=
1,2
,
3
I
f
max
⁡
⁡
f
f
i
,
i
=
1,2
,
3
>
0
.
Then I
t
-
1 is a subset of most informative attributes and the algorithm terminates. Otherwise go to Step 5.
Step
5. To determine the closest coordinates, calculate
(16)
D
0
=
min
⁡
x
*
1
r
-
x
*
2
r
:
r
ϵ
I
t
and define the following set:
(17)
R
t
=
r
ϵ
I
t
:
x
*
1
r
-
x
*
2
r
≤
D
0
+
ϵ
.
Step
6. Construct the set
(18)
I
t
+
1
=
I
t
∖
R
t
.
If I
t
+
1
=
∅, then I
t is the subset of most informative attributes. If I
t
+
1
=
1
,thenI
t
+
1 is the subset of most informative attributes. Then the algorithm terminates; otherwise set t
=
t
+
1 and go to Step 2.


## 4. Solving Optimization Problems

In this section, algorithm for solving problems as mentioned in the classification algorithm has been discussed. Since these functions are nonsmooth and estimate of subgradients is difficult, direct search methods of optimization seem to be the best option for solving them. The main attraction of direct search methods is their ability to find optimal solutions without the need for computing derivatives, in contrast to the more familiar gradient-based methods [24].Direct search algorithms can be applied for problems that are difficult to be solved with traditional optimization techniques, including problems that are difficult to model mathematically or are not well defined. They can be also applied when the objective function is discontinuous, stochastic, highly nonlinear, or undefined derivative.In general, direct search algorithms are called pattern search algorithms and both the generalized pattern search (GPS) algorithm and the MADS algorithm are pattern search algorithms that compute a sequence of points that get closer and closer to the optimal point. At each step, the algorithm investigates a set of points, called a mesh, around the current point (the point computed at the previous step of the algorithm). The mesh is created by adding the current point to a scalar multiple of a set of vectors called a pattern. If the pattern search algorithm discovers a point in the mesh that makes better (decreases) the objective function at the current point, the new point becomes the current point at the next step of the algorithm.


### 4.1. The MADS Method

MADS methods are designed to only use function values and require only a numerical value of the objective; no knowledge about the internal structure of the problem is needed. These methods can quickly and easily be used in nonlinear, nonconvex, nondifferentiable, discontinuous, or undermined problems [19]. The convergence analysis of MADS guarantees necessary optimality conditions of the first and second orders under certain assumptions [19]. A general optimization problem can be as follows:
(19)
Min
F
x
s
.
t
.
x
∈
X
,
where  F
:
R
→
R
n
∪
+
∞
,
X
=
x
∈
R
n
∣
C
i
(
x
)
≤
0
,
i
=
1
,
…
,
m
,
L
1
≤
x
≤
L
2
,
∁
:
R
n
→
R
m, and L
1
∈
(
{
-
∞
}
∪
R
)
n
,
L
2
∈
(
{
+
∞
}
∪
R
)
n.MADS is an iterative algorithm. Each iteration (shown by the subscriptk
) is initiated with the current best feasible solution x
k, known as the incumbent solution, and each iteration k of the MADS algorithm can be stated by two steps. First, an optional search step over the space of variables is performed as long as it is a finite process and all trial points lie on a mesh. If no better point is found or no global search is applied, the algorithm goes to a compulsory local exploration step (compulsory because it ensures convergence). Second is the poll step; at most 2
n trial mesh points near the incumbent solution are chosen (the poll set) and evaluated. If no better neighbor is found, the mesh is refined. If an improved mesh point x
k
+
1
ϵ
X is found, the mesh is kept the same or coarsened, and then x
k
+
1 is the next incumbent. The exploration directions vary at each iteration and become dense with probability 1. This is the main difference between the pattern search and MADS algorithms. General constraints can be handled with a barrier approach, which redefines the objective as in the following equation:
(20)
F
X
=
F
x
if
x
∈
X
+
∞
otherwise
.
Then, MADS is applied to the unconstrained barrier problem
(21)
Min
⁡
x
F
X
x
.
The feasible region X can be nonlinear, nonconvex, nondifferentiable, or disjoint. There are no hypotheses made on the domain, except that the initial point must be feasible. The convergence results depend on the local smoothness of F (and not F
X, which is obviously discontinuous on the boundary of X).Algorithm 3 (the MADS algorithm).
A general and flexible algorithmic framework for MADS is studied in [19]. This general framework is then specialized to a specific algorithmic implementation. The main steps of the algorithm are summarized as follows.
Step
1 (initialization). The user defines the starting point and the initial mesh size.
The algorithm initializes other parameters for subsequent steps.
Step
2 (request for an improved mesh point). Consider the following steps:(i)
global search (optional): evaluation ofF over a finite subset of points defined by the mesh;
(ii)
local poll (mandatory): definition of a poll set and evaluation ofF over points in that set.
Step
3 (parameters update). Parameters are updated.
Step
4 (termination). If some stopping criterion is reached, stop; if not, go back to Step 2.


## 4.1. The MADS Method

MADS methods are designed to only use function values and require only a numerical value of the objective; no knowledge about the internal structure of the problem is needed. These methods can quickly and easily be used in nonlinear, nonconvex, nondifferentiable, discontinuous, or undermined problems [19]. The convergence analysis of MADS guarantees necessary optimality conditions of the first and second orders under certain assumptions [19]. A general optimization problem can be as follows:
(19)
Min
F
x
s
.
t
.
x
∈
X
,
where  F
:
R
→
R
n
∪
+
∞
,
X
=
x
∈
R
n
∣
C
i
(
x
)
≤
0
,
i
=
1
,
…
,
m
,
L
1
≤
x
≤
L
2
,
∁
:
R
n
→
R
m, and L
1
∈
(
{
-
∞
}
∪
R
)
n
,
L
2
∈
(
{
+
∞
}
∪
R
)
n.MADS is an iterative algorithm. Each iteration (shown by the subscriptk
) is initiated with the current best feasible solution x
k, known as the incumbent solution, and each iteration k of the MADS algorithm can be stated by two steps. First, an optional search step over the space of variables is performed as long as it is a finite process and all trial points lie on a mesh. If no better point is found or no global search is applied, the algorithm goes to a compulsory local exploration step (compulsory because it ensures convergence). Second is the poll step; at most 2
n trial mesh points near the incumbent solution are chosen (the poll set) and evaluated. If no better neighbor is found, the mesh is refined. If an improved mesh point x
k
+
1
ϵ
X is found, the mesh is kept the same or coarsened, and then x
k
+
1 is the next incumbent. The exploration directions vary at each iteration and become dense with probability 1. This is the main difference between the pattern search and MADS algorithms. General constraints can be handled with a barrier approach, which redefines the objective as in the following equation:
(20)
F
X
=
F
x
if
x
∈
X
+
∞
otherwise
.
Then, MADS is applied to the unconstrained barrier problem
(21)
Min
⁡
x
F
X
x
.
The feasible region X can be nonlinear, nonconvex, nondifferentiable, or disjoint. There are no hypotheses made on the domain, except that the initial point must be feasible. The convergence results depend on the local smoothness of F (and not F
X, which is obviously discontinuous on the boundary of X).Algorithm 3 (the MADS algorithm).
A general and flexible algorithmic framework for MADS is studied in [19]. This general framework is then specialized to a specific algorithmic implementation. The main steps of the algorithm are summarized as follows.
Step
1 (initialization). The user defines the starting point and the initial mesh size.
The algorithm initializes other parameters for subsequent steps.
Step
2 (request for an improved mesh point). Consider the following steps:(i)
global search (optional): evaluation ofF over a finite subset of points defined by the mesh;
(ii)
local poll (mandatory): definition of a poll set and evaluation ofF over points in that set.
Step
3 (parameters update). Parameters are updated.
Step
4 (termination). If some stopping criterion is reached, stop; if not, go back to Step 2.


## 5. Results of Numerical Experiments

To verify the efficiency of the proposed algorithms a number of numerical experiments with real-world data sets have been carried out on a PC, Intel Core 2 Duo CPU, 1.95 GB of RAM.The Australian credit dataset, the breast cancer dataset, the diabetes dataset, the heart disease dataset, the liver-disorder dataset, the German Numer dataset, and the mushroom dataset have been applied in numerical experiments.The description of these datasets can be found in UCI Machine Learning Repository [25].In Table1, Nshows the number of samples of dataset, C presents the number of classes of dataset, and F is the number of features.Table 1
Properties of the examined databases.
Data
N
C
F
Australian credit
690
2
14
Breast cancer
569
2
30
Diabetes
768
2
8
Heart
303
2
13
Liver
345
2
6
German Numer
1,000
2
24
Mushrooms
8,124
2
112First, all features were normalized. This is done by a nonsingular matrix so that standard deviation values of all features are 1. In order to evaluate the performance, 10-fold cross-validation was used where a sample from each dataset was selected and then divided into 10 equal sized subsets. Next, a subset was selected and designated as the test set and the union of the remainder nine subsets was used as the training set. After the application of Algorithm2 which calculates the subset of informative attributes and selection of the features, the classification model was validated with the test subset. This process was repeated where each of the 10 subsets was successively selected as the test set. Accordingly, the proposed method was run 10 times and the classification accuracy rate was calculated by averaging across all 10 test runs.Note. In feature selection algorithm (Algorithm 2) T
i
,
i
=
1,2
,
3 (maximum numbers of added “bad points” in each iteration of the feature selection algorithm for each class of dataset) have important role in the execution of this algorithm and therefore in numerical experiments one or two percent of value of each class dataset for T
1
,
T
2, and T
3
=
T
1
+
T
2 have been consideredIn comparison withT
i
,
i
=
1,2
,
3 introduced in [18] results of numerical experiments have shown that this algorithm significantly reduces the number of attributes, so that 3 attributes were used in the diabetes dataset, the breast cancer dataset, the liver-disorder dataset, and the Australian credit dataset, 11 attributes in the heart disease dataset, 4 attributes in the German dataset, and 6 attributes in the mushroom dataset for solving classification problem. While in comparison with those obtained by the proposed method and the results obtained in [18] we can see that, for the Australian credit dataset, the number of features is decreased from 6 to 3; for the breast cancer dataset it was the same, while for the heart disease dataset, the number of features is increased from 3 to 11.In numerical experiments Algorithm1 was used for classification of datasets with 10-fold cross-validation and the MADS algorithm has been applied for solving problems in Algorithm 1; then in this research it is called MA and is supposed as ε
=
0.01. Results of the numerical experiments are presented in Tables 2–8. In Tables 2–8, e
train represents the error rate for the training data and e
test shows the error rate for the test data, that is, criteria for goodness of one method.Table 2
Results in the Australian credit dataset.
Algorithm
e
test
e
train
MA
7.3
15.4
NBTree
16.8
RBF
43.29
KStar
19.18
Ridor
12.65
VFI
16.47
MultiBoost
12.71
Bayes net
12.13
PSO
18.77
Different approaches from Michie
13.1
13.2Table 3
Results in the breast cancer database.
Algorithm
e
test
e
train
MA
2.5
2.9
PMM
3.5
1.4
RLP
2.8
2.3
HMM
2.6
2.1
NBTree
7.69
RBF
20.27
KStar
2.44
Ridor
6.36
VFI
7.34
MultiBoost
5.59
Bayes net
4.19
PSO
3.49Table 4
Results in the diabetes dataset.
Algorithm
e
test
e
train
MA
19.7
18.0
PMM
23.3
19.4
RLP
24.0
23.3
HMM
24.1
21.6
SVM
25.0
24.0
NBTree
25.52
RBF
39.16
KStar
34.05
Ridor
29.31
VFI
34.37
MultiBoost
27.08
Bayes net
25.52
PSO
21.77
IncNet
22.4
Fuzzy approach
22.4
FLEXNFIS
21.4
FNN
18.2
Different approaches from Michie
22.3Table 5
Results in the heart dataset.
Algorithm
e
test
e
train
MA
14.8
14.5
PMM
17.8
8.6
RLP
16.5
15.5
HMM
17.2
12.5
SVM
24.1
15.3
NBTree
22.36
RBF
45.25
KStar
26.70
Ridor
22.89
VFI
18.42
MultiBoost
18.42
Bayes net
18.42
PSO
15.73
Different approaches from Michie
37.4
35.1Table 6
Results in the liver dataset.
Algorithm
e
test
e
train
MA
27.8
23.0
PMM
31.6
25.1
RLP
33.1
31.0
HMM
33.4
27.8
NBTree
39.0
39.8
RULES-4
44.1
C4.5
34.5
Naïve Bayes
36.6
BNND
38.6
BNNF
38.2Table 7
Results in the German Numer dataset.
Algorithm
e
test
e
train
MA
25.53
24.7
PNN
31.71
C4.5
28.53
SMO
25.16
Boost
28.81
Bayes
25.66
DC-RBFNN
25.29
MLP
27.86
KNN
31.52
RIPPER
30.0
GPF
24.76
FFA
46.59
HS
44.76
PSO
40.48Table 8
Results in the mushroom dataset.
Algorithm
e
test
e
train
MA
0.27
0.26
PNN
0.29
C4.5
0
SMO
0
Boost
3.21
Bayes
4.46
DC-RBFNN
1.67
1.77
MLP
0
KNN
0.17
INB
4.7
RIPPER
0
FFA
0
HS
0.05
PSO
0.04Also the numerical results of the parametric misclassification minimization (PMM) [26], robust linear programming (RLP) [27], the hybrid misclassification minimization (HMM) [28], support vector machines algorithms [29], the K-nearest neighbor algorithm (KNN), the multilayer perceptron (MLP), the probabilistic neural network (PNN), and the sequential minimal optimization algorithm (SMO) [30, 31] were used for the purpose of comparison. Moreover the results obtained by particle swarm optimization algorithm (PSO) [32, 33], music-inspired harmony search algorithm (HS) [34], fire fly algorithm (FFA) [35] and its references, the Waikato Environment for Knowledge Analysis (WEKA) system release 3.4 [36], which contains a large number of such techniques that were divided into different groups, were equally used for comparison. From each of such groups, some representatives have been chosen. They are as follows: the radial basis function artificial neural network (RBF) [37], among the lazy, the KStar [38], among the rule-based ones the ripple down rule (Ridor) [39], and among others the voting feature interval (VFI) [40]. Similarly, we have the MultiBoostAB [41] and among the Bayesian the Bayes net [42]. Parameter values used for any technique are those set as default in WEKA. Also the results obtained by support vector machines algorithm [10], IncNet [43], fuzzy approach [44], FLEXNFIS [45], FNN [46], RULES-4 [47] and C4.5 [48], Naïve Bayes [49, 50], BNND and BNNF methods from [51], SSVM [52], RSVM [53], SVM [54], LSSVM [55], FAIRS [56], DC-RBFNN [57], Boost [58], RIPPER [59], INB [60], and GPF [61] were used in the experiments.The results of numerical experiments obtained by using 23 algorithms of classification from Michie [62], presented in Chapter 9 of this book, were also applied; these are statistical, neural network, and machine learning algorithms. In addition, only the best results obtained by these algorithms are presented in Tables 2–8.The results for the Australian credit database are presented in Table2, which indicates that the accuracy of the proposed method is higher than the accuracies of other methods pointed out in the table.The results for second database, breast cancer database, are presented in Table3. It shows that the accuracy of proposed method is higher than the accuracies of other methods except for KStar and HMM methods in which the accuracies are quite close to that of the proposed method.For the diabetes database, the results of numerical experiments are presented in Table4, which shows that the accuracy of proposed method is higher than the accuracies of other methods pointed out in this table except that of FNN method in which the accuracy is the best.The results for the heart database are presented in Table5. From these results and the previous results, it is safe to conclude that the accuracy of proposed method is the best and, thus, the most suitable for this dataset.The results for the liver database are presented in Table6 which shows that the accuracy of proposed method is better than the accuracies of other methods pointed out in the table except for PMM method in which the accuracy is the best. From these results and the previous results, it is safe to conclude that the accuracy of proposed method is the best and, thus, the most suitable for this dataset.The results for the German database are presented in Table7, which confirms that the errors of the proposed method are lower than the errors of other methods pointed out in the table, except for SMO, DC-RBFNN, and GPF methods in which the errors are lower than MA method.The results for the last database, mushroom database, are presented in Table8. It shows that the errors of the proposed method are near 0. These are showing goodness of the proposed method.As shown in Tables2–8, the MA model obtains the best or near the best prediction accuracies in almost all datasets.Further, in order to evaluate important factors in the performance of the MADS algorithm for solving the classification problem, different experiments were carried out on the datasets as mentioned earlier in this paper. Here only the main results obtained are presented from different experiments conducted; this is done to avoid unnecessary details for the sake of summary. In this research, mesh factorsΔ
ME
r
,
Δ
MC
r have been defined as: Δ
ME
r, is mesh contraction factor used when iteration is unsuccessful, Δ
ME
r is mesh expansion factor which expands mesh when iteration is successful and Δ
ME
r
=
1
/
2
r
,
Δ
ME
r
=
2
r,  r
∈
N. Also it was found that the e
test decreases when r increases and the best value for r is near 5.Since the direction of poll set is chosen as random in MADS algorithm, therefore each performance of this algorithm gives new result and so MA method was performed 10 times and average of solutions is presented in the following tables. Also it was found that the standard deviations of them (solutions) are near zero.Various experiments have been accomplished on the datasets as mentioned before in the classification algorithm with having different values ofε (so that ε
ϵ
[
1
,
10
-
2
]) for finding the best value for ε. Therefore, the good value was observed around 10
-
2. Also to appraise important factors in the performance of the MADS algorithm for solving classification problem, the same experiments have been done by different strategies have been made in the MADS algorithm for step search (that means (
1
) search step is empty, (
2
) when n
+
1 random direction was chosen for mesh set in search step, (
3
) when genetic algorithm was chosen for step search, and (
4
) when Nelder-Mead algorithm was chosen for step search); and the results were almost the same.Therefore, the results are presented in Tables2–8, show that AM gives good results compared with other methods for all datasets. The results of numerical experiments demonstrate that the proposed algorithms are effective for solving classification problems.


## 6. Conclusions

In this paper a new algorithm was proposed for solving classification problem where the algorithm includes the nonsmooth and nonconvex optimization problems. The new proposed algorithm is based on classes in the database which use cluster centers so that, for each class, the cluster analysis problem with more estimation is solved.The MADS method was used for solving the nonsmooth optimization problems. The new method was tested using real-world datasets. Results of these computational experiments show the effectiveness of the new algorithms. In the future, the size of datasets will increase; obviously applying feature selection is useful for classification problem and therefore it seems that the feature selection procedure should be further studied. Also proposing new globalization strategies for this method based on combining with other good methods similar to PSO for solving classification problems as future study is suggested.



---

*Source: 318478-2014-11-27.xml*