# Optimization ELM Based on Rough Set for Predicting the Label of Military Simulation Data

Authors: Xiao-jian Ding; Ming Lei
Journal: Mathematical Problems in Engineering
 (2014)

---

## Abstract

By combining rough set theory with optimization extreme learning machine (OELM), a new hybrid machine learning technique is introduced for military simulation data classification in this study. First, multivariate discretization method is implemented to convert continuous military simulation data into discrete data. Then, rough set theory is employed to generate the simple rules and to remove irrelevant and redundant variables. Finally, OELM is compared with classical extreme learning machine (ELM) and support vector machine (SVM) to evaluate the performance of both original and reduced military simulation datasets. Experimental results demonstrate that, with the help of RS strategy, OELM can significantly improve the testing rate of military simulation data. Additionally, OELM is less sensitive to model parameters and can be modeled easily.


---

## Body

## 1. Introduction

Information warfare, which is the competitive use of information in modern forces, has recently become of increasing importance to the military domain. Information superiority and knowledge dominance are key points to winning the future war. There has been a substantial growth in the amount of stored information in military systems, mainly due to the rapid development of military information construction. However, the large amount of stored data becomes impracticable for specialists to analyze through conventional methods.Military data research has gained great importance and interest in recent years, especially in operational areas. Data learning technology provides important analysis and decision services to aid the commander. To provide decision support for commanders, it is worth studying how to effectively learn the massive data that emerges in operations. The data used in the equipment war gaming system was classified on the basis of introducing terms about verification, validation, and certification (VV&C) and a sort criterion of authoritative data sources (ADS) was also put forward, as described by Liguo et al. [1]. Li et al. [2] analyzed the characteristics of military data, put forward methods to classify data based on their characteristics and functions, and discussed data sources for combat simulation. Gao et al. [3] introduced a military simulation data aggregation framework for extraction, transformation, transportation, and loading based on message oriented middleware. As seen in [4], preliminary study on simulation data mining by rough set (RS) theory was done. RS enables us to find the dependencies and to reduce the number of attributes contained in military simulation data. With the help of RS, valuable results can be obtained by mining the battle simulation data, and the attributes which make a strong impact on performance of datasets learning can be found. However, given newly generated battle data, one cannot predict the classification results with the method of [4].In this study, a new regularization algorithm-optimization extreme learning machine (OELM) [5], together with RS theory, is selected as main components for constructing an integrated predictor. OELM is an optimization algorithm based on conventional extreme learning machine (ELM) [6–11], which has the following properties: first, compared to the traditional ELM, the minimization norm of output weights enables OELM to get better generalization performance; second, compared to support vector machine (SVM) [12], OELM finds the optimal solution in the search space of [0,C]N, where SVM always searches the suboptimal solution of OELM due to equation constraints, so OELM usually achieves better generalization performance than a traditional SVM; third, OELM is less sensitive to specified parameters and can be implemented easily.In this paper, the performance of an integrated method of multivariate discretization, RS theory, and OELM for military simulation data analysis is investigated and compared with state-of-the-art classifiers: ELM and SVM. As there exist some interactions between different attributes of military simulation data, multivariate discretization attempts to find the correct cuts by taking into account one attribute independently from the others. RS theory is adopted to reduce redundant attributes of military simulation data, and then OELM algorithm is employed to train this new dataset with optimal subset attributes.We start this work with description of the integrated method in Section2. Military simulation data preprocessing method is discussed in Section 3. Military simulation data analysis is shown in Section 4. Section 5 presents the experimental setup and details the results arrived at. Finally, Section 6 gives the conclusion.


## 2. Integrated Method

In this section, RS theory is adopted to reduce redundant attributes of the military simulation dataset, and then OELM algorithm is employed to train this new dataset with optimal attributes subset. The main contributions of this work are as follows:(1) multivariate discretization method is used to preprocess the original military dataset; (2) rules generated by RS method are used to create a reduced military dataset; (3) OELM, together with two other popular algorithms, is used to measure the generalization performance of both original dataset and reduced dataset.


### 2.1. OELM Review

ForN arbitrary distinct samples {xi,ti}i=1N, where xi=[xi1,xi2,…,xid]∈Rd and ti∈{-1,1}, the decision function given by an ELM is
(1)f(x)=sign⁡(∑i=1LβiG(ai,bi,x))=sign⁡(β·h(x)),
where βi is the output weight from the ith hidden node to the output node and G(ai,bi,x) is the output of the ith hidden node. h(x)=[G(a1,b1,x),…,G(aL,bL,x)] is the output vector of the hidden layer, and it can map the training data xi from the input space to the L-dimensional ELM feature space.Based on the theory of Bartlett [13], feedforward neural network with smaller norm of weight, not only has smaller training error, but also obtains better generalization performance. Therefore, ELM amounts to minimizing the training error with the minimized norm of the output weight:
(2)Minimize:∑i=1N∥β·h(xi)-ti∥Minimize:∥β∥.Any set of training data transformed from the input space to the ELM feature space with the mappingh(x) is linearly separable. In order to prevent the classification problem of overfitting, variables ξi, i=1,…,N are introduced and one can minimize the testing error as follows:
(3)β·h(xi)≥1-ξiforti=+1,β·h(xi)≤-1+ξiforti=-1.
Two inequalities in (3) can be combined into one set of inequalities:
(4)ti·β·h(xi)≥1-ξi∀i.
Thus, OELM works based on solving the following quadratic program mathematical model:
(5)Minimize:12∥β∥2+C∑i=1NξiSubject to:ti·β·h(xi)≥1-ξi,i=1,…,Nξi≥0,i=1,…,N.Compared to SVM’s optimization problem, there are three differences between SVM and OELM:(1)
The mappingϕ(xi) used in SVM is usually unknown and cannot be computed directly. The function h(xi) used in OELM can be any bounded nonconstant piecewise continuous activation function for additive node, which can be calculated exactly.(2)
Kernel parameters used in SVM need to be tuned manually, whereas all parameters ofh(x) for OELM are selected randomly.(3)
The biasb is not needed in the constraints of OELM’s optimization problem, because the separating hyperplane of OELM tends to pass through the origin.


### 2.2. Karush-Kuhn-Tucker Conditions

According to Lagrange optimization theory, minimizing the norm of output weights is equivalent to solving the dual optimization problem of (5):
(6)Minimize:12∑i=1N∑j=1NtitjKELM(xi,xj)αiαj-∑i=1NαiSubject to:0≤αi≤C,i=1,…,N.
ELM kernel matrix can be defined as
(7)KELM(xi,xj)=h(xi)·h(xj).For the sake of convenience, (6) can be written compactly as
(8)Minimize:f(α)=12αΤKα-eΤαSubject to:0≤αi≤C,i=1,…,N,
where f:RN→R is a convex function, K=KELMtΤt∈RN×N is positive semidefinite matrix, α=[α1,…,αN]Τ∈RN, and e=[1,…,1]Τ∈RN.Letg(α)=∂f(α)/∂α=Kα-e. Let gi(α) denote the ith element of g(α); that is, gi(α)=[Kα-e]i. The Lagrange function of (8) is
(9)L(α,η,ρ)=f(α)-∑i=1Nηiαi-∑i=1Nρi(C-αi),
where ηi and ρi are the Lagrange multipliers and are nonnegative values.In order to find the optimal solutions of (9) we should have
(10)∂L(α,η,ρ)∂αi=0⟹gi(α)-ηi+ρi=0.
Based on the Karush-Kuhn-Tucker (KKT) theorem, the KKT conditions of (10) should beprimal feasibility(11)0≤αi≤C,∀i;dual feasibility(12)ηi≥0,∀i,(13)ρi≥0,∀i;complementary slackness(14)ηiαi=0,∀i,(15)ρi(C-αi)=0,∀i.Furthermore, consider the following.(1)
Ifαi=0, from (15), we have ρi=0. Thus, from (10) and (12), we have
(16)gi(α)≥0.(2)
If0<αi<C, from (14) and (15), we have ηi=0 and ρi=0. Thus, from (10), we have
(17)gi(α)=0.(3)
Ifαi=C, from (14), we have ηi=0. Thus, from (10) and (13), we have
(18)gi(α)≤0.The KKT conditions are both necessary and sufficient for optimality. Thus, (8) is solved when for all i(19)αi=0⟺gi(α)≥0,0<αi<C⟺gi(α)=0,αi=C⟺gi(α)≤0.Active set approach is the best choice to solve (8) because it is an iterative algorithm and it maintains feasibility of the vector α (i.e., 0≤αi≤C) while iteration continues until KKT conditions (19) are satisfied.


### 2.3. Rough Set

Rough set (RS) term, which was first introduced by Pawlak et al. [14] has become a popular pattern recognition tool for generating logical rules for prediction. Attribute reduction, which is one of the core parts of RS theory, means to find out the most informative attributes and remove the irrelevant or unimportant attributes with minimal information loss.RS is the approximation of an uncertain set by a pair of precise concepts called lower and upper approximations. The lower approximation comprises elements belonging to it, whereas the upper approximation of the set includes elements which are possible members of the set.RS model is defined in terms of an information system, which can be defined formally as 4-tuple in [15, 16]:
(20)S=〈U,Q,V,f〉,
where U={u1,u2,…,un} is a nonempty set of objects called the universe, Q={q1,q2,…,qn} is a finite set of attributes, Vq is the domain of attribute q, and V=⋃q∈QVq. f=U×Q→V is an information function that f(x,q)∈Vq for each q∈Q, x∈U. Each object of universe is described by a vector
(21)Desq(x)=[f(x,q1),f(x,q2),…,f(x,qm)],
where x∈U. To every nonempty subset of attributes P is associated with an indiscernibility relation on U, denoted by IP:
(22)IP={(x,y)∈U×U:f(x,q)=f(y,q),∀q∈P}.Equation (22) is an equivalence relation. The family of all the equivalence classes of the IP is denoted by U∣IP and class containing an element x by IP(x).Formally, letX be a nonempty set of U. Set X is approximated by means of P-lower and P-upper approximations of X:
(23)P_(X)={x∈U:IP(x)⊆X},P-(X)=⋃x∈XIP(x).
The P-boundary of X is denoted by Bn(X)(24)Bn(X)=P-(X)-P_(X).
The following relation holds: P_(X)⊆X⊆P-(X).There are three regions of interest, they are the inner region for objects inP_(X), the outer region for objects not in P-(X), and the boundary region for the objects that are uncertain.


## 2.1. OELM Review

ForN arbitrary distinct samples {xi,ti}i=1N, where xi=[xi1,xi2,…,xid]∈Rd and ti∈{-1,1}, the decision function given by an ELM is
(1)f(x)=sign⁡(∑i=1LβiG(ai,bi,x))=sign⁡(β·h(x)),
where βi is the output weight from the ith hidden node to the output node and G(ai,bi,x) is the output of the ith hidden node. h(x)=[G(a1,b1,x),…,G(aL,bL,x)] is the output vector of the hidden layer, and it can map the training data xi from the input space to the L-dimensional ELM feature space.Based on the theory of Bartlett [13], feedforward neural network with smaller norm of weight, not only has smaller training error, but also obtains better generalization performance. Therefore, ELM amounts to minimizing the training error with the minimized norm of the output weight:
(2)Minimize:∑i=1N∥β·h(xi)-ti∥Minimize:∥β∥.Any set of training data transformed from the input space to the ELM feature space with the mappingh(x) is linearly separable. In order to prevent the classification problem of overfitting, variables ξi, i=1,…,N are introduced and one can minimize the testing error as follows:
(3)β·h(xi)≥1-ξiforti=+1,β·h(xi)≤-1+ξiforti=-1.
Two inequalities in (3) can be combined into one set of inequalities:
(4)ti·β·h(xi)≥1-ξi∀i.
Thus, OELM works based on solving the following quadratic program mathematical model:
(5)Minimize:12∥β∥2+C∑i=1NξiSubject to:ti·β·h(xi)≥1-ξi,i=1,…,Nξi≥0,i=1,…,N.Compared to SVM’s optimization problem, there are three differences between SVM and OELM:(1)
The mappingϕ(xi) used in SVM is usually unknown and cannot be computed directly. The function h(xi) used in OELM can be any bounded nonconstant piecewise continuous activation function for additive node, which can be calculated exactly.(2)
Kernel parameters used in SVM need to be tuned manually, whereas all parameters ofh(x) for OELM are selected randomly.(3)
The biasb is not needed in the constraints of OELM’s optimization problem, because the separating hyperplane of OELM tends to pass through the origin.


## 2.2. Karush-Kuhn-Tucker Conditions

According to Lagrange optimization theory, minimizing the norm of output weights is equivalent to solving the dual optimization problem of (5):
(6)Minimize:12∑i=1N∑j=1NtitjKELM(xi,xj)αiαj-∑i=1NαiSubject to:0≤αi≤C,i=1,…,N.
ELM kernel matrix can be defined as
(7)KELM(xi,xj)=h(xi)·h(xj).For the sake of convenience, (6) can be written compactly as
(8)Minimize:f(α)=12αΤKα-eΤαSubject to:0≤αi≤C,i=1,…,N,
where f:RN→R is a convex function, K=KELMtΤt∈RN×N is positive semidefinite matrix, α=[α1,…,αN]Τ∈RN, and e=[1,…,1]Τ∈RN.Letg(α)=∂f(α)/∂α=Kα-e. Let gi(α) denote the ith element of g(α); that is, gi(α)=[Kα-e]i. The Lagrange function of (8) is
(9)L(α,η,ρ)=f(α)-∑i=1Nηiαi-∑i=1Nρi(C-αi),
where ηi and ρi are the Lagrange multipliers and are nonnegative values.In order to find the optimal solutions of (9) we should have
(10)∂L(α,η,ρ)∂αi=0⟹gi(α)-ηi+ρi=0.
Based on the Karush-Kuhn-Tucker (KKT) theorem, the KKT conditions of (10) should beprimal feasibility(11)0≤αi≤C,∀i;dual feasibility(12)ηi≥0,∀i,(13)ρi≥0,∀i;complementary slackness(14)ηiαi=0,∀i,(15)ρi(C-αi)=0,∀i.Furthermore, consider the following.(1)
Ifαi=0, from (15), we have ρi=0. Thus, from (10) and (12), we have
(16)gi(α)≥0.(2)
If0<αi<C, from (14) and (15), we have ηi=0 and ρi=0. Thus, from (10), we have
(17)gi(α)=0.(3)
Ifαi=C, from (14), we have ηi=0. Thus, from (10) and (13), we have
(18)gi(α)≤0.The KKT conditions are both necessary and sufficient for optimality. Thus, (8) is solved when for all i(19)αi=0⟺gi(α)≥0,0<αi<C⟺gi(α)=0,αi=C⟺gi(α)≤0.Active set approach is the best choice to solve (8) because it is an iterative algorithm and it maintains feasibility of the vector α (i.e., 0≤αi≤C) while iteration continues until KKT conditions (19) are satisfied.


## 2.3. Rough Set

Rough set (RS) term, which was first introduced by Pawlak et al. [14] has become a popular pattern recognition tool for generating logical rules for prediction. Attribute reduction, which is one of the core parts of RS theory, means to find out the most informative attributes and remove the irrelevant or unimportant attributes with minimal information loss.RS is the approximation of an uncertain set by a pair of precise concepts called lower and upper approximations. The lower approximation comprises elements belonging to it, whereas the upper approximation of the set includes elements which are possible members of the set.RS model is defined in terms of an information system, which can be defined formally as 4-tuple in [15, 16]:
(20)S=〈U,Q,V,f〉,
where U={u1,u2,…,un} is a nonempty set of objects called the universe, Q={q1,q2,…,qn} is a finite set of attributes, Vq is the domain of attribute q, and V=⋃q∈QVq. f=U×Q→V is an information function that f(x,q)∈Vq for each q∈Q, x∈U. Each object of universe is described by a vector
(21)Desq(x)=[f(x,q1),f(x,q2),…,f(x,qm)],
where x∈U. To every nonempty subset of attributes P is associated with an indiscernibility relation on U, denoted by IP:
(22)IP={(x,y)∈U×U:f(x,q)=f(y,q),∀q∈P}.Equation (22) is an equivalence relation. The family of all the equivalence classes of the IP is denoted by U∣IP and class containing an element x by IP(x).Formally, letX be a nonempty set of U. Set X is approximated by means of P-lower and P-upper approximations of X:
(23)P_(X)={x∈U:IP(x)⊆X},P-(X)=⋃x∈XIP(x).
The P-boundary of X is denoted by Bn(X)(24)Bn(X)=P-(X)-P_(X).
The following relation holds: P_(X)⊆X⊆P-(X).There are three regions of interest, they are the inner region for objects inP_(X), the outer region for objects not in P-(X), and the boundary region for the objects that are uncertain.


## 3. Military Simulation Data Preprocessing

The performance of the integrated method is evaluated on countermeasure simulation data of tank formation [4], which was exercised on medium undulating ground, hills, and Taiwan highland for digital armored battalion simulation system. The specification of the dataset is presented in Table 1.Table 1
Parameters of simulation data.
W1
T1
S
V
W2
T2
F
P
R
1
Fine
Flat
Fast
More
Small
Short
Poor
Normal
High
2
Fine
Undulating
Slow
More
Small
Short
Poor
Normal
Low
3
Fog
Flat
Slow
Less
Small
Long
Better
Normal
High
4
Fog
Flat
Fast
Less
Big
Long
Better
Nervous
Low
5
Fine
Undulating
Slow
Less
Big
Long
Better
Normal
High
6
Fine
Undulating
Fast
Less
Big
Short
Poor
Nervous
Low
7
Fine
Flat
Slow
More
Small
Short
Better
Nervous
High
8
Fog
Flat
Slow
More
Small
Short
Poor
Nervous
Low
9
Fine
Undulating
Fast
Less
Big
Short
Better
Nervous
High
10
Fog
Undulating
Fast
Less
Small
Long
Better
Normal
Low
11
Fine
Flat
Fast
Less
Small
Short
Better
Normal
High
12
Fog
Flat
Fast
More
Big
Short
Poor
Normal
LowFrom Table1, items “W1,” “T1,” …, and “R” represent “Weather,” “Terrain,” “Speed,” “Vegetation,” “Wind,” “Time,” “Formation,” “Psychology,” and “Shooting rate,” respectively. ELM algorithm can be applied only to datasets composed of categorical attributes but attributes in Table 1 are continuous variables. Discretization process is important for char variables because it is less prone to variance in estimation from small fragmented data and it provides better performance for rule extraction. The main discretization process methods are supervised [17], hierarchical [18], top-down [19], and multivariate [20]. The last one is employed in our work.Multivariate discretization quantifies simultaneously many features. In the preprocessing phase, attribute “Weather” is converted into two new attributes “fine” and “fog,” which is shown in Table2.Table 2
Converted attribute of “Weather.”
Attribute
Fine
Fog
Weather = fine
1
0
Weather = fog
0
1Based on examples shown in Table2, other attributes are converted in Table 3.Table 3
Discretization dataset.
W1
T1
S
V
W2
T2
F
P
R
1
0
1
0
1
0
1
0
1
0
0
1
0
1
1
0
1
1
0
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
0
1
1
0
0
1
0
1
1
0
1
0
1
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
1
0
0
1
0
1
0
0
1
0
1
0
1
0
1
1
0
1
0
1
0
1
1
0
0
1
1
0
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
0
1
1
0
1
0
0
1
1
0
0
1
1
0
1
1
0
0
1
1
0
1
0
0
1
0
1
0
1
0
1
0
0
1
1
0
0
1
0
1
0
1
1
0
0
1
1
0
1
0
1
1
0
0
1
1
0
1
0
1
0
1
0
0
1
0
1
0
1
0
0
1
1
0
0
1
1
0
1
0
1
0
1
1
0
1
0
1
0
0
1
0
1
0
1
1
0
0


## 4. Military Simulation Data Analysis

Generally speaking, it is a data mining task to distinguish the probability of shooting rate. The goal of military data analysis is to predict the unknown value of shooting rate, such as high or low. More specifically, military simulation data analysis process can be modeled as a classification problem.A typical military dataset is sparse compared to a traditional classification dataset. It usually contains small samples but with many features. This kind of sparseness often descends the performance of some classifiers. To improve the classification rate, one can remove the irrelevant or redundant features. Intuitively speaking, a classifier trained in lower-dimensional feature space is expected to capture the inherent data distribution and performs better in such feature space.There are many advantages of variable and feature selection, such as facilitating data visualization and data understanding, reducing training and testing time, and so forth. We give a 3-dimensional visualization of dataset discussed in Section4, as seen in Figure 1.Figure 1
Military simulation data visualization.In Figure1, only 4 samples can be seen. As there are 12 samples for entire dataset, we conjecture that some samples overlap each other. It can be obviously concluded that much redundant information is contained in this dataset.RS method is a well-established method for feature selection. It is widely used in many applications and more flexible than other methods, such as principal component analysis (PCA) [21]. One can use RS method to discover the data dependencies and reduce the number of attributes. For comparison purpose we used five other widely used feature selection methods (i) PCA, (ii) multidimensional scaling (MDS), (iii) generalized discriminant snalysis (GDA), (iv) factor analysis (FA), and (v) Isomap.After feature selection, more or less overlapping phenomenon appears in Figure2, besides the “Rough set” subfigure. The main goal of feature selection method is to choose a subset of useful variables, which exclude many redundant, but relevant variables. As seen in Figure 2, RS method obtains the most relevant variables, because “Rough set” subfigure has the most overlapping region. The usefulness of these variables is directly judged by the estimated accuracy of the learning method. Learning method based on features chosen with RS method achieves error rates lower than other methods.Figure 2
Comparison of 6 feature selection methods.


## 5. Performance Evaluation

In this section we report the performance of the proposed integrated method both on original and reduced military simulation datasets and compare the performance with other state-of-the-art classifiers. The reduced dataset is prepossessed using the procedure described in Section4. After converting, 8 attributes in the original data are scattered to 16 attributes. In our classification application, the inputs (attributes) are normalized into the range [0,1], while the outputs (targets) are normalized into the range [-1,1]. All the simulations are running in the MATLAB R2008a (Windows version) environments with Intel 3.0 GHZ and 2 G RAM.For comparison purpose we used two other widely used classification methods, ELM and SVM. For the implementation of SVM, we use the Spider library for MATLAB, which is publicly available from [22]. Both MATLAB codes of ELM and OELM can be downloaded from ELM host site: http://www.ntu.edu.sg/home/egbhuang/. To train SVM, radial basis function (RBF) kernel K(xi,xj)=exp⁡(-γ∥xi-xj∥2) is used in our experiments. ELM with sigmoidal activation function G(a,b,x)=1/(1+exp⁡(-(a·x+b))) is tested. Both the Gaussian RBF activation function G(a,b,x)=exp⁡(-∥x-a∥2/b) and the sigmoidal additive activation function have been used in the simulation of OELM. For both ELM and OELM with sigmoidal additive activation function and RBF activation function, the input weights a and biases b are randomly generated from (-1,1)N×(0,1) based on the uniform probability distribution.


### 5.1. Reduced Dataset

According to the RS strategy in [4], the number of attributes is reduced from 16 to 8 on military simulation dataset, which is shown in Table 4.Table 4
Reduced military simulation dataset.
W1
T1
S
F
R
1
0
1
0
1
0
0
1
1
0
1
1
0
0
1
1
0
1
1
0
0
1
0
1
1
0
1
1
0
1
0
0
1
1
0
1
1
0
0
1
1
0
1
0
1
1
0
1
0
1
0
1
0
1
1
0
0
1
0
1
0
1
0
0
1
1
0
1
0
1
0
0
1
0
0
1
1
0
0
1
0
0
1
1
0
0
1
0
1
0
0
1
0
1
1
0
1
0
0
0
1
1
0
1
0
0
1
0


### 5.2. Selection of Parameters

Two parameters for RBF kernel of SVM, the cost constantC and the kernel parameter γ, are both sensitive to the generalization performance. A straightforward way for model selection is the grid search strategy [23]. According to the method of [24], two parameters are tuned on a grid space of {10−3, 10−2, 0.05, 10−1, 0.2, 0.5, 1, 2, 5, 10, 20, 50, 102, 103, 104} × {10−3, 10−2, 10−1, 0.2, 0.4, 0.8, 1, 2, 5, 10, 20, 50, 102, 103, 104}. Therefore, for each problem we try 15 × 15 = 225 combinations of parameters (C,γ) for SVM. Figure 3 shows the generalization performance of SVM for each combination.The performance of SVM for 225 combinations.
(a)
Original dataset
(b)
Reduced datasetAs can be seen in Figure3, the specific C and kernel parameter γ value form a grid in the (C,γ) parameter space. It is clear that the wider the search space is, the more possibility the grid search method finds the best parameter combination. However, the grid search method exhaustively searches the grid to find a best combination. Figure 3(a) indicates that the best generalization performance is achieved for 18 combinations on original dataset, and the performance of SVM highly depends on the combination of parameters. Similar results are obtained in Figure 3(b).For ELM, there is only one parameterL (optimal number of hidden nodes) that needs to be determined. During our simulation it is found that ELM with 5 hidden nodes can generally obtain good performance for both datasets. Generally speaking, the generalization performance of OELM is not sensitive to the number of hidden nodes L [5]. We set the number of hidden nodes 1, 2, 3, 4, 5, and 6. Moreover, there is one more parameter that needs to be determined, which is the number of cost parameter C. Similar to [5], we set this parameter 10−3, 10−2, 0.05, 10−1, 0.2, 0.5, 1, 2, 5, 10, 20, 50, 102, 103, and 104. The optimal parameter(s) L(L,C) are selected to obtain the best testing performance. Table 5 shows the optimal parameters of three classifiers on two datasets.Table 5
Comparison between three ELM, SVM, and OELM for two datasets.
Datasets
Classifiers
L /(L, C)/(C,γ)
Training  time (s)
TR (%)
Dev (%)
Original dataset
ELM
5
2.6243
e
-
004
55.00
3.85
SVM
(1,10-2)
0.0079
59.33
2.67
OELM-S
(5,102)
0.0013
65.12
2.89
OELM-R
(5,102)
8.5874
e
-
004
63.87
2.95
Reduced  dataset
ELM
5
2.7404
e
-
004
60.33
2.79
SVM
(1,10-2)
0.0077
60.00
1.85
OELM-S
(5,103)
0.0018
74.67
2.06
OELM-R
(6,102)
9.8123
e
-
004
72.92
2.36


### 5.3. Performance Comparison on Two Datasets

We evaluate the performance of three classifiers by repeated random splitting; that is, two datasets are equally partitioned into one training set and one test set. To avoid selection bias, there are at least one positive sample (labeled as “1”) and one negative sample (labeled as “−1”) in the training set. Such splitting is repeated 50 times. Training time, testing rate (TR) averaged over the 50 trials, and the corresponding standard deviations (Dev) are reported.Table5 draws a comparison between three classifiers, as well as some comparative results on different kernels. All of results represent the average, taken over 50 trials. OELM-S and OELM-R mean OELM using Sigmoidal and RBF kernel, respectively. As observed from Table 5, the performance of OELM-S and that of OELM-R is similar to each other except that OELM-S requires twice training time taken by OELM-R. We can see that the training time taken by ELM is much less than other classifiers, and the worst performance is also obtained by ELM. The performance of OELM-S or OELM-R is obviously higher than other classifiers. In order to further study this phenomenon, an extra experiment is conducted in Table 6.Table 6
Random permutation performance comparison of ELM, SVM, and OELM-S.
Trials
Random permutation(training set)
TR (%)
Original dataset
Reduced dataset
ELM
SVM
OELM-S
ELM
SVM
OELM-S
1
(1, 8, 10, 2, 3, 12)
50.00
33.33
66.67
66.67
33.33
83.33
2
(3, 11, 6, 9,10, 5)
50.00
33.33
50.00
50.00
33.33
66.67
3
(3, 9, 6, 11, 8, 10)
83.33
100
83.33
83.33
100
100
4
(8, 3, 11, 1, 10, 5)
50.00
33.33
50.00
66.67
33.33
83.33
5
(9, 6, 3, 1, 8, 4)
50.00
100
66.67
50.00
100
83.33
6
(9, 8, 12, 7, 1, 6)
50.00
100
66.67
66.67
100
83.33
7
(5, 9, 7, 10, 3, 6)
50.00
33.33
33.33
66.67
33.33
66.67
8
(5, 2, 4, 10, 6, 8)
33.33
16.67
16.67
33.33
16.67
50.00
9
(3, 5, 11, 8, 10, 12)
66.67
100
66.67
83.33
100
100
10
(5, 2, 7, 12, 9, 1)
50.00
33.33
50.00
50.00
33.33
66.67Seen from Table6, among 10 trials, the number of training and test set is fixed according to the above setting, but the order of training set is randomly shuffled for each trial. Although 100% testing rate is obtained for SVM on 4 trials, testing rate on other trials is no more than 33.33%. These results seem very unstable due to the randomness. Interestingly, OELM-S runs more stable than the other two classifiers, and this is why the performance of OELM-S is much higher than SVM and ELM.Strictly speaking, it seems that, after data reduction, the performance of SVM cannot be improved on all trials. Moreover, OELM-S is exactly a good choice employed in our integrated strategy for improving the generalization performance of military simulation data.


## 5.1. Reduced Dataset

According to the RS strategy in [4], the number of attributes is reduced from 16 to 8 on military simulation dataset, which is shown in Table 4.Table 4
Reduced military simulation dataset.
W1
T1
S
F
R
1
0
1
0
1
0
0
1
1
0
1
1
0
0
1
1
0
1
1
0
0
1
0
1
1
0
1
1
0
1
0
0
1
1
0
1
1
0
0
1
1
0
1
0
1
1
0
1
0
1
0
1
0
1
1
0
0
1
0
1
0
1
0
0
1
1
0
1
0
1
0
0
1
0
0
1
1
0
0
1
0
0
1
1
0
0
1
0
1
0
0
1
0
1
1
0
1
0
0
0
1
1
0
1
0
0
1
0


## 5.2. Selection of Parameters

Two parameters for RBF kernel of SVM, the cost constantC and the kernel parameter γ, are both sensitive to the generalization performance. A straightforward way for model selection is the grid search strategy [23]. According to the method of [24], two parameters are tuned on a grid space of {10−3, 10−2, 0.05, 10−1, 0.2, 0.5, 1, 2, 5, 10, 20, 50, 102, 103, 104} × {10−3, 10−2, 10−1, 0.2, 0.4, 0.8, 1, 2, 5, 10, 20, 50, 102, 103, 104}. Therefore, for each problem we try 15 × 15 = 225 combinations of parameters (C,γ) for SVM. Figure 3 shows the generalization performance of SVM for each combination.The performance of SVM for 225 combinations.
(a)
Original dataset
(b)
Reduced datasetAs can be seen in Figure3, the specific C and kernel parameter γ value form a grid in the (C,γ) parameter space. It is clear that the wider the search space is, the more possibility the grid search method finds the best parameter combination. However, the grid search method exhaustively searches the grid to find a best combination. Figure 3(a) indicates that the best generalization performance is achieved for 18 combinations on original dataset, and the performance of SVM highly depends on the combination of parameters. Similar results are obtained in Figure 3(b).For ELM, there is only one parameterL (optimal number of hidden nodes) that needs to be determined. During our simulation it is found that ELM with 5 hidden nodes can generally obtain good performance for both datasets. Generally speaking, the generalization performance of OELM is not sensitive to the number of hidden nodes L [5]. We set the number of hidden nodes 1, 2, 3, 4, 5, and 6. Moreover, there is one more parameter that needs to be determined, which is the number of cost parameter C. Similar to [5], we set this parameter 10−3, 10−2, 0.05, 10−1, 0.2, 0.5, 1, 2, 5, 10, 20, 50, 102, 103, and 104. The optimal parameter(s) L(L,C) are selected to obtain the best testing performance. Table 5 shows the optimal parameters of three classifiers on two datasets.Table 5
Comparison between three ELM, SVM, and OELM for two datasets.
Datasets
Classifiers
L /(L, C)/(C,γ)
Training  time (s)
TR (%)
Dev (%)
Original dataset
ELM
5
2.6243
e
-
004
55.00
3.85
SVM
(1,10-2)
0.0079
59.33
2.67
OELM-S
(5,102)
0.0013
65.12
2.89
OELM-R
(5,102)
8.5874
e
-
004
63.87
2.95
Reduced  dataset
ELM
5
2.7404
e
-
004
60.33
2.79
SVM
(1,10-2)
0.0077
60.00
1.85
OELM-S
(5,103)
0.0018
74.67
2.06
OELM-R
(6,102)
9.8123
e
-
004
72.92
2.36


## 5.3. Performance Comparison on Two Datasets

We evaluate the performance of three classifiers by repeated random splitting; that is, two datasets are equally partitioned into one training set and one test set. To avoid selection bias, there are at least one positive sample (labeled as “1”) and one negative sample (labeled as “−1”) in the training set. Such splitting is repeated 50 times. Training time, testing rate (TR) averaged over the 50 trials, and the corresponding standard deviations (Dev) are reported.Table5 draws a comparison between three classifiers, as well as some comparative results on different kernels. All of results represent the average, taken over 50 trials. OELM-S and OELM-R mean OELM using Sigmoidal and RBF kernel, respectively. As observed from Table 5, the performance of OELM-S and that of OELM-R is similar to each other except that OELM-S requires twice training time taken by OELM-R. We can see that the training time taken by ELM is much less than other classifiers, and the worst performance is also obtained by ELM. The performance of OELM-S or OELM-R is obviously higher than other classifiers. In order to further study this phenomenon, an extra experiment is conducted in Table 6.Table 6
Random permutation performance comparison of ELM, SVM, and OELM-S.
Trials
Random permutation(training set)
TR (%)
Original dataset
Reduced dataset
ELM
SVM
OELM-S
ELM
SVM
OELM-S
1
(1, 8, 10, 2, 3, 12)
50.00
33.33
66.67
66.67
33.33
83.33
2
(3, 11, 6, 9,10, 5)
50.00
33.33
50.00
50.00
33.33
66.67
3
(3, 9, 6, 11, 8, 10)
83.33
100
83.33
83.33
100
100
4
(8, 3, 11, 1, 10, 5)
50.00
33.33
50.00
66.67
33.33
83.33
5
(9, 6, 3, 1, 8, 4)
50.00
100
66.67
50.00
100
83.33
6
(9, 8, 12, 7, 1, 6)
50.00
100
66.67
66.67
100
83.33
7
(5, 9, 7, 10, 3, 6)
50.00
33.33
33.33
66.67
33.33
66.67
8
(5, 2, 4, 10, 6, 8)
33.33
16.67
16.67
33.33
16.67
50.00
9
(3, 5, 11, 8, 10, 12)
66.67
100
66.67
83.33
100
100
10
(5, 2, 7, 12, 9, 1)
50.00
33.33
50.00
50.00
33.33
66.67Seen from Table6, among 10 trials, the number of training and test set is fixed according to the above setting, but the order of training set is randomly shuffled for each trial. Although 100% testing rate is obtained for SVM on 4 trials, testing rate on other trials is no more than 33.33%. These results seem very unstable due to the randomness. Interestingly, OELM-S runs more stable than the other two classifiers, and this is why the performance of OELM-S is much higher than SVM and ELM.Strictly speaking, it seems that, after data reduction, the performance of SVM cannot be improved on all trials. Moreover, OELM-S is exactly a good choice employed in our integrated strategy for improving the generalization performance of military simulation data.


## 6. Conclusions

This paper has introduced an integrated learning method of RS and OELM for military simulation dataset. The generalization performance of OELM is less sensitive to the user specified parameters especially for the number of hidden nodes. Compared to SVM, one can use OELM easily and effectively without parameter(s) tuning process. Significantly better results for the reduced dataset are obtained by employing OELM classifier over crisp discretization. This method has the potential for further military application because it can provide real-time decision support for commanders in battlefield.



---

*Source: 706178-2014-09-25.xml*