# Deep Network Based on Stacked Orthogonal Convex Incremental ELM Autoencoders

Authors: Chao Wang; Jianhui Wang; Shusheng Gu
Journal: Mathematical Problems in Engineering
 (2016)

---

## Abstract

Extreme learning machine (ELM) as an emerging technology has recently attracted many researchers’ interest due to its fast learning speed and state-of-the-art generalization ability in the implementation. Meanwhile, the incremental extreme learning machine (I-ELM) based on incremental learning algorithm was proposed which outperforms many popular learning algorithms. However, the incremental algorithms with ELM do not recalculate the output weights of all the existing nodes when a new node is added and cannot obtain the least-squares solution of output weight vectors. In this paper, we propose orthogonal convex incremental learning machine (OCI-ELM) with Gram-Schmidt orthogonalization method and Barron’s convex optimization learning method to solve the nonconvex optimization problem and least-squares solution problem, and then we give the rigorous proofs in theory. Moreover, in this paper, we propose a deep architecture based on stacked OCI-ELM autoencoders according to stacked generalization philosophy for solving large and complex data problems. The experimental results verified with both UCI datasets and large datasets demonstrate that the deep network based on stacked OCI-ELM autoencoders (DOC-IELM-AEs) outperforms the other methods mentioned in the paper with better performance on regression and classification problems.


---

## Body

## 1. Introduction

Extreme learning machine (ELM) proposed by Huang et al. [1, 2] is a specific type of single-hidden layer feedforward network (SLFN) with randomly generated additive or RBF hidden nodes and hidden node parameters, which has recently been extensively studied by many researchers in various areas of scientific research and engineering due to the excellent approximation capability. Wang et al. presented ASLGEM-ELM algorithm, which provides some useful guidelines for improving the generalization ability of SLFNs trained with ELM [3]. Alongside probing deeply into the research of theory and its application, ELM has become one of the leading trends for fast learning [4–7]. Recently, Huang et al. [8] have proposed an algorithm called incremental extreme learning machine (I-ELM) which randomly adds nodes to the hidden layer one by one and freezes the output weights of the existing hidden nodes when a new hidden node is added [9–12]. Then, Huang et al. [13] also showed its universal approximation capability for the case of fully complex hidden nodes. I-ELM is fully automatically implemented and in theory no intervention is required for the learning process from users. But there still exist some issues to be tackled [14]:(1)
The redundant nodes can be generated in I-ELM, which have a minor effect on the outputs of the network. Moreover, the existence of redundant nodes can eventually increase the complexity of the network.
(2)
The convergence rate of I-ELM is slower than ELM, and the number of hidden nodes in I-ELM is sometimes larger than the dimension of samples for the training.In this paper, we propose a method called orthogonal convex extreme learning machine (OCI-ELM) to further settle the aforementioned problems of I-ELM. With the rigorous proofs in theory, we can obtain the least-squares solution ofH
β
=
T and faster convergence rate by adopting the Gram-Schmidt orthogonalization method incorporated into CI-ELM [15]. The simulations on real-world datasets show that the proposed OCI-ELM algorithm can achieve faster convergence rates, more compact neural network, and better generalization performance than both I-ELM and the improved I-ELM algorithms while keeping the simplicity and efficiency of ELM.Recently, deep learning has attracted many research interests with its remarkable success in many applications [16–18]. Deep learning is an artificial neural network learning algorithm which has multilayer perceptrons. Deep learning has achieved an approximation of complex functions and alleviated the optimization difficulty associated with the deep models [19–21]. Motivated by the remarkable success of deep learning [22, 23], we propose a new stacked architecture to solve large and complex data problems using OCI-ELM autoencoder as the training algorithm in each layer, which incorporates the excellent performance of OCI-ELM with the ability of complex function approximation derived from deep architectures. We implemented OCI-ELM autoencoder in each iteration of deep orthogonal convex incremental extreme machine (DOC-IELM) to reconstruct the input data and estimate the errors of the prediction functions with the scheme of layer-by-layer architectures. Both the supervised and the unsupervised data all can be the pertaining input of the new proposed deep network. Moreover, the OCI-ELM autoencoder-based deep network (DOC-IELM-AEs) can suffice to achieve the efficiency improvement for generalization performance.To show the effectiveness of DOC-IELM-AEs, we apply it to both the ordinary real-world datasets with UCI datasets and large datasets with MNIST, OCR Letters, NORB, and USPS datasets. The simulations show that the proposed deep model possesses better accuracy of testing and more compact network architecture than the aforementioned improved I-ELM and other deep models without incurring the out-of-memory problem.This paper is organized as follows. Section2 reviews the preliminary knowledge of incremental extreme learning machine (I-ELM). Section 3 describes OCI-ELM algorithm, the proposed model which adopts the Gram-Schmidt orthogonalization method into convex I-ELM (CI-ELM). Section 4 makes a comparison between OCI-ELM and other algorithms. Section 5 presents the details of DOC-IELM-AEs algorithm and compares the performance with deep architecture models. Section 6 applies the DOC-IELM-AEs algorithm into elongation prediction of strips. Finally, Section 7 concludes this paper.


## 2. Related Works

In this section, the main concepts and theory of the I-ELM [8] algorithm are shortly reviewed. For the sake of generality, we assume that the network has only one linear output node, and all the analysis can be easily extended into multinonlinear output nodes cases. Consider a training dataset ℵ
=
{
(
x
i
,
t
i
)
∣
x
i
∈
R
n
,
t
i
∈
R
m
,
1
≤
i
≤
N
}; the SLFN with L additive hidden nodes and activation function σ
(
x
) can be represented by(1)
f
n
x
=
∑
i
=
1
L
β
i
σ
a
i
·
x
j
+
b
j
,
j
=
1,2
,
…
,
N
,where a
i
=
[
a
i
1
,
a
i
2
,
…
,
a
i
n
]
T is the weight vector connecting the input layer to the ith hidden node, β
i
=
[
β
i
1
,
β
i
2
,
…
,
β
i
n
]
T is the weight connecting the ith hidden node to the output node, b
i is the threshold of the ith hidden node, and σ is the hidden node activation function.The I-ELM proposed by Huang et al. is different from the conventional ELM algorithm; I-ELM is an automatic algorithm which can randomly add hidden nodes to the network one by one and freeze all the weights of the existing hidden nodes when a new hidden node is added, until the expected learning accuracy is obtained or the maximum number of hidden nodes is reached. Thus, I-ELM algorithm can be summarized in Algorithm1.Algorithm 1 (incremental extreme learning machine (I-ELM)).
Given a training datasetℵ
=
{
(
x
i
,
t
i
)
∣
x
i
∈
R
n
,
t
i
∈
R
m
,
1
≤
i
≤
N
}, activation function g
(
x
), number of hidden nodes L, expected learning accuracy ε, and maximum number of hidden nodes L
m
a
x, one has the following.
Step 1 (initialization). Let L
=
0 and residual error E
=
t, where t
=
[
t
1
,
t
2
,
…
,
t
N
]
T.
Step 2 (learning step). While L
<
L
m
a
x, E
>
ε,(a)
increase the number of hidden nodes by one;
(b)
assign random input weightw and bias b for hidden nodes L;
(c)
calculate the residual error after adding the new hidden node;
(d)
calculate the output weightβ for the new hidden nodes: β
=
E
,
H
L
/
H
L
2;
(e)
calculate the residual error:E
=
E
-
β
·
H
L;Endwhile.


## 3. The Proposed Orthogonal Convex Incremental Extreme Learning Machine (OCI-ELM)

The motivation for the work in this section comes from the important properties of basic ELM as follows:(1)
The special solutionβ
^
=
H
†
T is one of the least-squares solutions of a general linear system H
β
=
T, meaning that the smallest training error can be reached by this special solution: H
β
^
-
T
=
H
H
†
T
-
T
=
min
β
⁡
H
β
-
T.
(2)
The smallest norm of weights: the special solutionβ
^
=
H
†
T has the smallest norm among all the least-squares solutions of H
β
=
T:(2)
β
^
=
H
†
T
<
β
∀
β
∈
β
:
H
β
-
T
≤
H
z
-
T
,
∀
z
∈
R
N
~
×
N
.
(3)
The minimum norm least-squares solution ofH
β
=
T is unique, which is β
^
=
H
†
T.In this section, we propose an improved I-ELM algorithm (OCI-ELM) based on Gram-Schmidt orthogonalization method combined with Barron’s convex optimization learning method and prove the OCI-ELM algorithm in theory which can obtain the least-squares solution ofH
β
=
T. Meanwhile, OCI-ELM can achieve a more compact network architecture, faster convergence rate, and better generalization performance than other improved I-ELM algorithms while retaining the I-ELM’s simplicity and efficiency.Theorem 2.
Gram-Schmidt orthogonalization process converts linearly independent vectors into orthogonal vectors [24]. Given a linearly independent vector set [
α
1
,
α
2
,
…
,
α
n
] in the inner product space V
⊆
R
n, the vector set β
1
,
β
2
,
…
,
β
n for Gram-Schmidt orthogonalization process is as follows [25]:(3)
β
1
=
α
1
,
β
2
=
α
2
-
β
1
,
α
2
β
1
,
β
1
β
1
,
β
3
=
α
3
-
β
1
,
α
3
β
1
,
β
1
β
1
-
β
2
,
α
3
β
2
,
β
2
β
2
,
β
k
=
α
k
-
β
1
,
α
k
β
1
,
β
1
β
1
-
β
2
,
α
k
β
2
,
β
2
β
2
-
⋯
-
β
k
-
1
,
α
k
β
k
-
1
,
β
k
-
1
β
k
-
1
,where α
1
,
α
2
,
…
,
α
n is the set of standardized vectors and β
1
,
β
2
,
…
,
β
n form an orthogonal set with the same linear span. For each index k
=
1,2
,
…
,
n, s
p
a
n
(
β
1
,
β
2
,
…
,
β
n
)
=
s
p
a
n
(
α
1
,
α
2
,
…
,
α
n
).Theorem 3.
Given an orthogonal vector set[
v
1
,
v
2
,
…
,
v
k
-
1
] in the inner product space V
⊆
R
n, if vector g
k can be expressed as a linear representation of v
1
,
v
2
,
…
,
v
k
-
1, one has(4)
v
k
=
g
k
-
v
1
,
g
k
v
1
,
v
1
v
1
-
v
2
,
g
k
v
2
,
v
2
v
2
-
⋯
-
v
k
-
1
,
g
k
v
k
-
1
,
v
k
-
1
v
k
-
1
=
0
.Proof.
Given the vector setv
1
,
v
2
,
…
,
v
k
-
1 and vector g
k, suppose there exist scalars c
1
,
c
2
,
…
,
c
k
-
1; then the linear combination of those vectors with those scalars as coefficients is(5)
g
k
=
c
1
v
1
+
c
2
v
2
+
⋯
+
c
k
-
1
v
k
-
1
.Substituting (4) into (5), we have v
k
=
0:(6)
β
n
=
e
n
-
1
,
g
n
-
f
n
-
1
g
n
-
f
n
-
1
2
.CI-ELM was originally proposed by Huang and Chen [15], which incorporates Barron’s convex optimization learning method into I-ELM. By recalculating the output weights of the existing hidden nodes randomly generated after a new node is added, the CI-ELM can obtain better performance than I-ELM. Incorporated with Gram-Schmidt orthogonalization and Barron’s convex optimization learning method, the process of OCI-ELM algorithm can be described in Algorithm 4.Algorithm 4 (orthogonal convex incremental extreme learning machine (OCI-ELM)).
Given a training datasetℵ
=
{
(
x
i
,
t
i
)
∣
x
i
∈
R
n
,
t
i
∈
R
m
,
i
=
1,2
,
…
,
N
}, where x
i
=
[
x
i
1
,
x
i
2
,
…
,
x
i
n
]
T and t
i
=
[
t
i
1
,
t
i
n
,
…
,
t
i
m
], and given activation function σ
(
x
), maximum number of iterations K
m
a
x, and expected learning accuracy ε, one has the following.
Step 1 (initialization). Let the number of initial hidden nodes L
=
0, the number of iterations K
=
0, and residual error E
=
t, where t
=
[
t
1
,
t
2
,
…
,
t
N
]
T.
Step 2. This step consists of two steps as follows.
Orthogonalization Step. In this step, the following is carried out:(
a
)
Increase the number of hidden nodesL and K by one, respectively: L
=
L
+
1 and K
=
K
+
1.
(
b
)
Randomly assign hidden node parameters(
a
L
,
b
L
) for new hidden node L and calculate the output g
k,(7)
v
k
=
g
k
-
v
1
,
g
k
v
1
,
v
1
v
1
-
v
2
,
g
k
v
2
,
v
2
v
2
-
⋯
-
v
k
-
1
,
g
k
v
k
-
1
,
v
k
-
1
v
k
-
1
,
and the hidden layer output matrixV,(8)
V
=
V
1
σ
w
1
·
x
1
+
b
1
⋯
V
m
σ
w
m
·
x
1
+
b
m
⋮
⋯
⋮
V
1
σ
w
1
·
x
n
+
b
1
⋯
V
m
σ
w
m
·
x
n
+
b
m
n
×
m
.
Learning Step. While L
1
<
L
m
a
x, E
>
ε,(
c
)
calculate the output weightβ
k for the newly added hidden node:(9)
β
k
=
E
·
E
-
F
-
V
k
T
E
-
F
-
V
k
·
E
-
F
-
V
k
T
;
(
d
)
recalculate the output weight vectors of all existing hidden nodes ifL
>
1:(10)
β
l
=
1
-
β
k
β
l
,
i
=
1,2
,
…
,
L
-
1
;
(
e
)
calculate the residual error after adding the new hidden nodeL:(11)
E
=
1
-
β
k
E
+
β
k
F
-
V
k
;Endwhile.The rigorous proof on the conclusion is detailedly discussed where OCI-ELM can obtain the least-squares solution ofH
β
=
T.Theorem 5.
Given a training datasetℵ
=
{
(
x
i
,
t
i
)
∣
x
i
∈
R
n
,
t
i
∈
R
m
,
i
=
1,2
,
…
,
N
} and number of hidden nodes L, where x
i
=
[
x
1
,
x
2
,
…
,
x
i
n
]
T and t
i
=
[
t
1
,
t
2
,
…
,
t
i
m
]
T, the hidden layer output matrix is G
L
=
[
g
1
,
g
2
,
…
,
g
L
]
∈
R
n
×
L, and the matrix of the output weights from the hidden nodes to the output nodes is β
L
=
[
β
1
,
β
2
,
…
,
β
L
]
T
∈
R
L. Let e
n
=
e
n
-
1
-
β
n
(
g
n
-
f
n
-
1
) denote the residual error function, and e
0
=
T, β
k
=
{
β
∗
∣
m
i
n
e
k
}
=
{
β
∗
∣
m
i
n
f
-
(
(
1
-
β
k
)
)
f
k
-
1
+
β
k
g
k
} holds with probability one if e
L
⊥
s
p
a
n
{
g
1
,
g
2
,
…
,
g
L
} and β
k
=
e
k
-
1
,
g
k
-
f
k
-
1
/
g
k
-
f
k
-
1
2 for all k
=
1,2
,
…
,
L.Proof.
The proof consists of two steps:(a)
Firstly, we provee
L
⊥
s
p
a
n
{
g
1
,
g
2
,
…
,
g
L
}.
(b)
And then, we further proveβ
k
=
{
β
∗
∣
m
i
n
f
-
(
(
1
-
β
k
)
)
f
k
-
1
+
β
k
g
k
}.(a) According to the condition given above, we have the following:(1)
Here,(12)
e
1
,
g
1
=
T
-
β
1
g
1
,
g
1
=
T
,
g
1
-
β
-
1
g
1
,
g
1
=
0
.
(2)
When the output weightβ
=
e
1
,
g
2
-
f
1
/
g
2
-
f
1
2, we have(13)
e
2
,
g
1
=
f
-
f
2
,
g
1
=
e
1
,
β
2
g
2
-
f
1
,
g
1
=
e
1
,
g
1
β
2
g
2
-
f
1
,
g
1
=
e
1
,
g
1
-
e
1
,
g
2
-
f
1
g
2
-
f
1
2
g
2
-
f
1
,
g
1
=
e
1
,
g
1
-
e
1
,
g
1
=
0
.
(3)
When the output weightβ
=
e
1
,
g
2
-
f
1
/
g
2
-
f
1
2, we also have(14)
e
2
,
g
2
=
f
-
f
2
,
g
2
=
e
1
-
β
2
g
2
-
f
1
,
g
2
=
e
1
,
g
2
-
e
1
,
g
2
-
f
1
g
2
-
f
1
2
g
2
-
f
1
,
g
2
=
e
1
,
g
2
-
e
1
,
g
2
=
0
.
(4)
When the output weightβ
k
=
e
k
-
1
,
g
k
-
f
k
-
1
/
g
k
-
f
k
-
1
2, suppose that, for all k
≤
l, we have(15)
e
k
,
g
k
=
e
k
-
1
-
β
k
g
k
-
f
k
-
1
,
g
k
=
e
k
-
1
,
g
k
-
β
k
g
k
-
f
k
-
1
,
g
k
=
e
k
-
1
,
g
k
-
e
k
-
1
,
g
k
-
f
k
-
1
g
k
-
f
k
-
1
2
g
k
-
f
k
-
1
,
g
k
=
e
k
-
1
,
g
k
-
e
k
-
1
,
g
k
=
0
.
(5)
When the output weightβ
k
=
e
k
-
1
,
g
k
-
f
k
-
1
/
g
k
-
f
k
-
1
2, suppose that, for all j
≤
k
-
1, we have(16)
e
k
,
g
j
=
e
k
-
1
-
β
k
g
k
-
f
k
-
1
,
g
j
=
e
k
-
1
,
g
j
-
β
k
g
k
-
f
k
-
1
,
g
j
=
e
k
-
1
,
g
j
-
e
k
-
1
,
g
k
-
f
k
-
1
g
k
-
f
k
-
1
2
g
k
-
f
k
-
1
,
g
j
=
e
k
-
1
,
g
j
-
e
k
-
1
,
g
j
=
0
.So, e
k
⊥
s
p
a
n
{
g
1
,
g
2
,
…
,
g
k
}; that is, e
L
⊥
s
p
a
n
{
g
1
,
g
2
,
…
,
g
L
}. Therefore,(17)
G
L
T
T
-
G
L
1
-
β
L
β
∗
=
0
.(b) According to (17), we have (
β
L
-
β
∗
)
T
G
L
T
(
T
-
G
L
(
1
-
β
L
)
β
∗
)
=
0, where β
∗
∈
R
L
×
1 is arbitrary; then, we have(18)
T
-
G
L
1
-
β
L
β
∗
2
≤
T
-
G
L
1
-
β
L
β
L
2
+
G
L
β
L
-
β
=
T
-
G
L
1
-
β
L
β
L
+
G
L
β
L
-
β
∗
=
T
-
G
L
β
∗
-
β
L
·
β
L
2
,
T
-
G
L
1
-
β
L
β
∗
=
T
-
G
L
β
∗
-
β
L
·
β
L
,
β
∗
-
β
L
·
β
∗
=
β
∗
-
β
L
·
β
L
,
β
L
=
β
∗
.And T
-
G
L
(
1
-
β
L
)
β
∗
=
T
-
G
L
(
1
-
β
L
)
β
L holds only if β
L
=
β
∗. Therefore, β
k
=
{
β
∗
∣
m
i
n
f
-
(
(
1
-
β
k
)
)
f
k
-
1
+
β
k
g
k
}.


## 4. Experiments and Analysis

In this section, we tested the generalization performance of the proposed OCI-ELM with other similar learning algorithms on ten UCI real-world datasets, including five regression and five classification problems, as shown in Table1. The simulations are conducted in MATLAB 2013a environment running on Windows 7 machine with 32 GB of memory and i7-990X (3.46 GHz) processor.Table 1
Specification of 10 benchmark problems.
Datasets
Training sample
Testing sample
Attributes
Type
Auto MPG
182
165
8
Regression
California Housing
8,800
8,260
8
Regression
Servo
92
60
4
Regression
Concrete Compressive Strength
975
830
9
Regression
Parkinsons
4,780
5,120
26
Regression
Delta Ailerons
5,080
4,600
6
Classification
Waveform II
3,300
2,800
40
Classification
Abalone
3,900
3,670
8
Classification
Breast Cancer
539
490
10
Classification
Energy Efficiency
740
600
8
ClassificationThe experimental results between OCI-ELM and some other ELM algorithms on regression and classification problems are given in Tables2 and 3. In Tables 2 and 3, the best results obtained by the OCI-ELM and the other 4 algorithms are italicized and shown in boldface. In Section 4.1, we compare the generalization performance of OCI-ELM with another six state-of-the-art algorithms on regression problems. In Section 4.2, we compare the generalization performance of OCI-ELM with the same six algorithms on classification problems. All these results in this section are obtained from thirty trials for all cases, and the mean results (mean), root-mean-square errors (RMSE), and standard deviations (Std.) are listed in the corresponding tables, respectively. The seven representative evolutionary algorithms are listed as follows:(i)
Convex incremental extreme learning machine (CI-ELM) [15].
(ii)
Parallel chaos search based incremental extreme learning machine (PC-ELM) [26].
(iii)
Leave-one-out incremental extreme learning machine (LOO-IELM) [27].
(iv)
Sparse Bayesian extreme learning machine (SB-ELM) [28].
(v)
Improved incremental regularized extreme learning machine (II-RELM) [11].
(vi)
Enhancement incremental regularized extreme learning machine (EIR-ELM) [12].Table 2
The comparisons of training and testing on the regression cases.
Datasets
Approaches
RMSE (training & testing)
⁢Hidden nodes & average time
Nodes (fixed)
Training
Testing
# nodes
Time (s)
Auto MPG (0.08)
CI-ELM(
2007
)
20
0.1043
0.1035
66.29
0.1485
PC-ELM(
2012
)
20
0.1014
0.1012
34.07
0.2783
LOO-IELM(
2014
)
20
0.1106
0.1104
49.91
0.3183
SB-ELM(
2014
)
20
0.1376
0.2307
≈130
0.0717
II-RELM(
2015
)
20
0.0998
0.1005
44.17
0.3483
EIR-ELM(
2016
)
20
0.0893
0.1005
31.05
0.3283
OCI-ELM
20
0.0827
0.0823
23.62
0.2204
California Housing (0.12)
CI-ELM
150
0.1601
0.1583
330.09
1.0051
PC-ELM
150
0.1389
0.1377
199.34
0.9810
LOO-IELM
150
0.1376
0.1374
217.08
0.9766
SB-ELM
150
0.1363
0.1369
—
—
II-RELM
150
0.1341
0.1339
192.33
0.9713
EIR-ELM
150
0.1274
0.1268
184.67
1.0017
OCI-ELM
150
0.1272
0.1263
172.15
0.9704
Servo (0.115)
CI-ELM
100
0.1428
0.1419
182.63
0.0806
PC-ELM
100
0.1373
0.1364
160.82
0.0701
LOO-IELM
100
0.1371
0.1368
155.72
0.0765
SB-ELM
100
0.1257
0.1254
≈127
0.0355
II-RELM
100
0.1303
0.1307
157.12
0.0886
EIR-ELM
100
0.1265
0.1264
147.80
0.0794
OCI-ELM
100
0.1238
0.1232
143.56
0.0828
CCS (0.035)
CI-ELM
150
0.0611
0.0602
229.86
0.5893
PC-ELM
150
0.0381
0.0365
162.79
1.1236
LOO-IELM
150
0.0372
0.0369
159.04
0.9427
SB-ELM
150
0.0366
0.0368
≈170
0.0872
II-RELM
150
0.0361
0.0363
163.82
0.8341
EIR-ELM
150
0.0348
0.0351
145.78
0.6305
OCI-ELM
150
0.0332
0.0346
130.04
0.5835
Parkinsons (0.14)
CI-ELM
250
0.0913
0.0906
170.02
3.3403
PC-ELM
250
0.0471
0.0463
63.55
4.7503
LOO-IELM
250
0.0453
0.0462
59.78
4.4452
SB-ELM
250
0.0388
0.0391
—
—
II-RELM
250
0.0389
0.0383
77.19
4.4189
EIR-ELM
250
0.0344
0.0347
48.92
3.9836
OCI-ELM
250
0.0301
0.0283
39.92
3.0819Table 3
The comparisons of training and testing on the classification cases.
Datasets
Approaches
Testing accuracy
⁢Hidden nodes & average time
Nodes (fixed)
Mean (%)
Std.
# nodes
Time (s)
Delta Ailerons (0.035)
CI-ELM(
2007
)
250
83.29
0.0036
369.32
1.3505
PC-ELM(
2012
)
250
90.02
0.0016
35.19
0.6829
LOO-IELM(
2014
)
250
91.17
0.0027
41.12
0.7761
SB-ELM(
2014
)
250
91.66
0.0071
≈220
0.0556
II-RELM(
2015
)
250
91.18
0.0042
51.16
0.7425
EIR-ELM(
2016
)
250
92.03
0.0019
34.29
1.1304
OCI-ELM
250
92.84
0.0012
31.73
0.7021
Waveform II (0.04)
CI-ELM
100
84.47
0.0182
200.11
3.0977
PC-ELM
100
89.81
0.0104
47.63
3.0954
LOO-IELM
250
88.93
0.0097
46.44
3.3437
SB-ELM
250
80.69
0.0181
—
—
II-RELM
250
90.64
0.0112
44.33
3.6603
EIR-ELM
250
91.15
0.0096
38.91
3.2267
OCI-ELM
100
93.11
0.0083
29.54
3.0864
Abalone (0.05)
CI-ELM
150
82.72
0.0022
150.37
0.4930
PC-ELM
150
93.57
0.0016
24.62
0.6177
LOO-IELM
250
90.51
0.0033
38.43
0.7102
SB-ELM
250
86.03
0.0107
≈180
0.0413
II-RELM
250
92.10
0.0034
35.74
0.6924
EIR-ELM
250
93.91
0.0018
24.16
0.8533
OCI-ELM
150
94.24
0.0015
21.41
0.6802
Breast Cancer (0.07)
CI-ELM
200
90.06
0.0145
88.30
0.0804
PC-ELM
200
93.23
0.0082
34.79
0.0992
LOO-IELM
250
93.07
0.0104
40.82
0.1102
SB-ELM
250
94.41
0.0075
≈150
0.0275
II-RELM
250
92.58
0.0095
55.27
0.1032
EIR-ELM
250
94.76
0.0078
31.18
0.1174
OCI-ELM
200
94.73
0.0067
34.58
0.1061
Energy Efficiency (0.055)
CI-ELM
150
91.78
0.0013
61.09
0.2966
PC-ELM
150
96.53
0.0008
41.08
0.3617
LOO-IELM
250
95.18
0.0024
27.94
0.3517
SB-ELM
250
92.16
0.0033
≈150
0.0658
II-RELM
250
95.29
0.0011
46.83
0.3826
EIR-ELM
250
96.67
0.0012
22.42
0.4011
OCI-ELM
150
97.25
0.0008
18.49
0.3979


### 4.1. Performance Comparison of Regression Problems

In this section, datasets Auto MPG, California Housing, Servo, CCS (Concrete Compressive Strength), and Parkinsons are conducted for the regression problems. Table2 shows the RMSE of the training and testing with fixed hidden nodes obtained from OCI-ELM and another six algorithms, respectively. Meanwhile, hidden nodes and learning time with the same stop RMSE are also shown in Table 2. For California Housing dataset in the table, the OCI-ELM provides lower training and testing RMSE rate (0.1272 and 0.1263) than CI-ELM (0.1601 and 0.1583), PC-ELM (0.1389 and 0.1377), LOO-IELM (0.1376 and 0.1374), SB-ELM (0.1363 and 0.1369), II-RELM (0.1341 and 0.1339), and EIR-ELM (0.1274 and 0.1268) with fixed nodes (20). For the stop criterion of RMSE 0.12, OCI-ELM also exhibits more compact network architecture with 127.15 nodes and faster speed with 0.9704 s, and the nodes and training speed of other algorithms are, respectively, 330.09 and 1.0051; 199.34 and 0.9810; 217.08 and 0.9766; and 192.33 and 0.9713. Where SB-ELM is the fixed ELM, thus there is a difficulty in finding the accurate stop criterion. Likewise, in the CCS dataset, the hidden node of SB-ELM is an approximate value; meanwhile, OCI-ELM shows better generalization performance than other algorithms in comparisons. Although, in the cases of Auto MPG, Servo, and Parkinsons, the learning time consumed by OCI-ELM shows that the presented algorithm is not the top spot, the average convergence rate of five regression problems consumed by OCI-ELM is still the fastest. Moreover, the average convergence rate demonstrates that the stability performance of OCI-ELM is better than other algorithms. The proposed algorithm can retain the simplicity and efficiency of incremental ELM and obtain the least-squares solution of H
β
=
T by incorporating the Gram-Schmidt orthogonalization method. The optimal solution obtained from H
β
=
T means that the best hidden node parameter leading to the largest residual error decreasing will be added to the existing network. Therefore, OCI-ELM can efficiently reduce the network complexity and meanwhile enhance the generalization performance of the algorithm.


### 4.2. Performance Comparison of Classification Problems

In this section, datasets Delta Ailerons, Waveform II, Abalone, Breast Cancer, and Energy Efficiency are conducted for the classification problems. Table3 shows the comparisons of the classification performance conducted on 5 UCI datasets of classification problems. With the same fixed hidden nodes listed in Table 3, the results of comparisons obtained from OCI-ELM are better than those of the other algorithms. For Waveform II dataset in the table, the OCI-ELM proposed displays better training accuracy and standard deviation of 93.11% and 0.0083 than CI-ELM (84.47% and 0.0182), PC-ELM (89.81% and 0.0104), LOO-IELM (88.93% and 0.0097), SB-ELM (80.69% and 0.0181), II-RELM (90.64% and 0.0112), and EIR-ELM (91.15% and 0.0096), thanks to the better classification ability of OCI-ELM. In addition, the hidden nodes 29.54 and the average time 3.0864 s are also less than others, which can demonstrate that OCI-ELM has more reasonable network structure than CI-ELM, PC-ELM, LOO-IELM, SB-ELM, II-RELM, and EIR-ELM, which efficiently reduces the complexity of the network. Although SB-ELM shows the advantages in training speed as the fixed ELM, OCI-ELM generally produces better performance in comprehensive consideration of accuracy and speed for practical problems which need higher accuracy demand.In short, OCI-ELM can generally achieve better performance on these regression and classification problems in terms of training (and testing) RMSE for regression and testing accuracy for classification. Moreover, the compactness of network and convergence rate also display the good performance of OCI-ELM algorithm.


## 4.1. Performance Comparison of Regression Problems

In this section, datasets Auto MPG, California Housing, Servo, CCS (Concrete Compressive Strength), and Parkinsons are conducted for the regression problems. Table2 shows the RMSE of the training and testing with fixed hidden nodes obtained from OCI-ELM and another six algorithms, respectively. Meanwhile, hidden nodes and learning time with the same stop RMSE are also shown in Table 2. For California Housing dataset in the table, the OCI-ELM provides lower training and testing RMSE rate (0.1272 and 0.1263) than CI-ELM (0.1601 and 0.1583), PC-ELM (0.1389 and 0.1377), LOO-IELM (0.1376 and 0.1374), SB-ELM (0.1363 and 0.1369), II-RELM (0.1341 and 0.1339), and EIR-ELM (0.1274 and 0.1268) with fixed nodes (20). For the stop criterion of RMSE 0.12, OCI-ELM also exhibits more compact network architecture with 127.15 nodes and faster speed with 0.9704 s, and the nodes and training speed of other algorithms are, respectively, 330.09 and 1.0051; 199.34 and 0.9810; 217.08 and 0.9766; and 192.33 and 0.9713. Where SB-ELM is the fixed ELM, thus there is a difficulty in finding the accurate stop criterion. Likewise, in the CCS dataset, the hidden node of SB-ELM is an approximate value; meanwhile, OCI-ELM shows better generalization performance than other algorithms in comparisons. Although, in the cases of Auto MPG, Servo, and Parkinsons, the learning time consumed by OCI-ELM shows that the presented algorithm is not the top spot, the average convergence rate of five regression problems consumed by OCI-ELM is still the fastest. Moreover, the average convergence rate demonstrates that the stability performance of OCI-ELM is better than other algorithms. The proposed algorithm can retain the simplicity and efficiency of incremental ELM and obtain the least-squares solution of H
β
=
T by incorporating the Gram-Schmidt orthogonalization method. The optimal solution obtained from H
β
=
T means that the best hidden node parameter leading to the largest residual error decreasing will be added to the existing network. Therefore, OCI-ELM can efficiently reduce the network complexity and meanwhile enhance the generalization performance of the algorithm.


## 4.2. Performance Comparison of Classification Problems

In this section, datasets Delta Ailerons, Waveform II, Abalone, Breast Cancer, and Energy Efficiency are conducted for the classification problems. Table3 shows the comparisons of the classification performance conducted on 5 UCI datasets of classification problems. With the same fixed hidden nodes listed in Table 3, the results of comparisons obtained from OCI-ELM are better than those of the other algorithms. For Waveform II dataset in the table, the OCI-ELM proposed displays better training accuracy and standard deviation of 93.11% and 0.0083 than CI-ELM (84.47% and 0.0182), PC-ELM (89.81% and 0.0104), LOO-IELM (88.93% and 0.0097), SB-ELM (80.69% and 0.0181), II-RELM (90.64% and 0.0112), and EIR-ELM (91.15% and 0.0096), thanks to the better classification ability of OCI-ELM. In addition, the hidden nodes 29.54 and the average time 3.0864 s are also less than others, which can demonstrate that OCI-ELM has more reasonable network structure than CI-ELM, PC-ELM, LOO-IELM, SB-ELM, II-RELM, and EIR-ELM, which efficiently reduces the complexity of the network. Although SB-ELM shows the advantages in training speed as the fixed ELM, OCI-ELM generally produces better performance in comprehensive consideration of accuracy and speed for practical problems which need higher accuracy demand.In short, OCI-ELM can generally achieve better performance on these regression and classification problems in terms of training (and testing) RMSE for regression and testing accuracy for classification. Moreover, the compactness of network and convergence rate also display the good performance of OCI-ELM algorithm.


## 5. Deep Network Based on Stacked OCI-ELM Autoencoders (DOC-IELM-AEs)

### 5.1. OCI-ELM Autoencoder

As an artificial neural network model, the autoencoder is frequently applied in the deep architecture approaches. Autoencoder is a kind of unsupervised neural network, where the input of network is equal to the output. Kasun et al. [29] proposed an autoencoder based on ELM (ELM-AE). According to their ELM-AE theory, the model of ELM-AE is composed of input layer, hidden layer, and output layer. In addition, the weights and biases of the hidden nodes are randomly generated via orthogonalization, and the input data is projected to a different or equal dimension space [30]; the expressions are as follows:(19)
h
=
σ
a
·
x
+
b
,
a
T
a
=
1
,
b
T
b
=
1
,
h
x
i
V
=
x
i
T
,
i
=
1,2
,
…
,
N
,where a
=
[
a
1
,
…
,
a
L
]
T are the weights generated orthogonally randomly and b
=
[
b
1
,
…
,
b
L
]
T are the biases generated orthogonally randomly between the input and hidden nodes. There are three calculation approaches to obtain the output weight β of ELM-AE: (1)
For sparse ELM-AE representations, output weightsβ can be calculated as follows:(20)
β
=
I
C
+
H
T
H
-
1
H
T
X
.
(2)
For compressed ELM-AE representations, output weightsβ can be calculated as follows:(21)
β
=
H
T
I
C
+
H
T
H
-
1
X
.
(3)
For equal dimension ELM-AE representations, output weightsβ can be calculated as follows:(22)
β
=
H
-
1
X
.In this section, we present the OCI-ELM, which is incorporated with Barron’s convex optimization learning method and Gram-Schmidt orthogonalization method into I-ELM to achieve the optimal least-squares solution as the training algorithm for an autoencoder instead of conventional autoencoders, which apply backpropagation algorithm (BP) for training to obtain the identity function and normal ELM for training the autoencoder. Because of the adoption of incremental algorithm, there is no need to set the number of hidden nodes according to the experience. With the initialization of the maximum valueL
m
a
x of hidden nodes the, number of hidden nodes can be increased by more than one node each time, until the stop criterions are met; for example, residual error E is equal to the expected learning accuracy ε or the number of hidden nodes L achieves L
m
a
x.As shown in Figure1, the model structure of OCI-ELM-AE can randomly control the number of the nodes without the computation accuracy. Given a training dataset ℵ
=
{
(
x
i
,
t
i
)
∣
x
i
∈
R
n
,
t
i
∈
R
m
,
i
=
1
,
…
,
N
}, where x
i
=
[
x
i
1
,
x
i
2
,
…
,
x
i
n
]
T and t
i
=
[
t
i
1
,
t
i
2
,
…
,
t
i
m
]
T, and given activation function σ
(
x
) and maximum number of hidden nodes in single layer L
m
a
x, the input data is reconstructed at the output layer through the following function:(23)
∑
i
=
1
l
β
i
σ
a
i
·
x
j
+
b
j
=
x
j
,
j
=
1,2
,
…
,
L
.The output weight can be obtained with the following:(24)
β
1
L
=
E
·
E
-
F
-
V
1
L
T
E
-
F
-
V
1
L
·
E
-
F
-
V
1
L
T
,
β
1
i
=
1
-
β
1
L
β
1
i
,
i
=
1,2
,
…
,
L
-
1
,where a
i
=
[
a
i
1
,
a
i
2
,
…
,
a
i
n
]
T is the input weight generated randomly and x
i
=
[
x
i
1
,
x
i
2
,
…
,
x
i
n
]
T are the input and output of the OCI-ELM-AE.Figure 1
Network diagram of OCI-ELM-AE.


### 5.2. Implementation of Stacked OCI-ELM Autoencoders in Deep Network

In 2006, Hinton et al. [31] presented the concept of deep learning to solve the problems of unsupervised data. Deep belief nets (DBNs) are probabilistic generative models which are first trained only with unlabeled data and then fine-tuned in a supervised mode. And then, another kind of deep network based on Restricted Boltzmann Machine (RBM) [32], deep Boltzmann machine (DBM) [33], was introduced by Salakhutdinov and Larochelle. The base building block of DBN is RBM. The ML-ELM was presented by Kasun et al. in 2013 [29]. There is no difference between the ML-ELM and the other deep learning models; ML-ELM performs layerwise unsupervised learning to train the parameters with the hidden layer weights which are initialized with ELM-AE, and the ML-ELM does not need to be fine-tuned. The AE-S-ELMs was proposed by Zhou et al. in 2014 [34]. The network consists of multiple ELMs with a small number of hidden nodes in each layer to substitute a single ELM with a large number of hidden nodes, and it implements ELM autoencoder in each iteration of S-ELMs algorithm to further improve the testing accuracy, especially for the unstructured large data without properly selected features.Algorithm 6 (deep network based on stacked orthogonal convex incremental ELM autoencoders (DOC-IELM-AEs)).
Given a training datasetℵ
=
{
(
x
i
,
t
i
)
∣
x
i
∈
R
n
,
t
i
∈
R
m
,
i
=
1,2
,
…
,
N
}, where x
i
=
[
x
i
1
,
x
i
2
,
…
,
x
i
n
]
T and t
i
=
[
t
i
1
,
t
i
n
,
…
,
t
i
m
], and given activation function σ
(
x
), maximum number of hidden nodes in single layer L
m
a
x, maximum number of iterations K
m
a
x, and expected learning accuracy ε, one has the following.Step 1 (initialization). Let the number of initial hidden nodes L
1
→
N
~
=
0, the number of iterations K
1
→
N
~
=
0, and residual error E
=
t, where t
=
[
t
1
,
t
-
2
,
…
,
t
N
]
T.Step 2 (orthogonal convex I-ELM autoencoder on layer 1). This step consists of two steps as follows.Orthogonalization Step. In this step, the following is carried out:(
a
)
Increase the number of hidden nodesL
1 and K
1 by one, respectively: L
1
=
L
1
+
1 and  K
1
=
K
1
+
1.
(
b
)
Randomly assign hidden node parameters(
a
L
,
b
L
) for new hidden node L and calculate the output g
k,(25)
v
k
=
g
k
-
v
1
,
g
k
v
1
,
v
1
v
1
-
v
2
,
g
k
v
2
,
v
2
v
2
-
⋯
-
v
k
-
1
,
g
k
v
k
-
1
,
v
k
-
1
v
k
-
1
,
and the hidden layer output matrix,(26)
V
=
V
1
σ
a
1
·
x
1
+
b
1
⋯
V
m
σ
a
m
·
x
1
+
b
m
⋮
⋯
⋮
V
1
σ
a
1
·
x
n
+
b
1
⋯
V
m
σ
a
m
·
x
n
+
b
m
n
×
m
.
Learning Step. While L
1
<
L
m
a
x, E
>
ε,(
c
)
calculate the output weightβ
1
(
L
) for the newly added hidden node: (27)
β
1
L
=
E
·
E
-
F
-
V
1
L
T
E
-
F
-
V
1
L
·
E
-
F
-
V
1
L
T
;
(
d
)
recalculate the output weight vectors of all existing hidden nodes ifL
1
>
1: (28)
β
1
i
=
1
-
β
1
L
β
1
i
,
i
=
1,2
,
…
,
L
-
1
;
(
e
)
calculate the residual error after adding the new hidden nodeL
1:(29)
E
=
1
-
β
1
L
E
+
β
1
L
F
-
V
1
L
.
Step 3 (orthogonal convex I-ELM autoencoder on layer 2
→
N
~). This step is carried out as follows.Learning Step. While L
2
→
N
~
<
L
m
a
x, E
>
ε,(a)
calculate the output weightβ
1
(
L
) for the newly added hidden node with the hidden layer output matrix V:(30)
β
2
→
N
~
L
=
E
·
E
-
F
-
V
2
→
N
~
L
T
E
-
F
-
V
2
→
N
~
L
·
E
-
F
-
V
2
→
N
~
L
T
;
(b)
recalculate the output weight vectors of all existing hidden nodes ifL
2
→
N
~
>
1:(31)
β
2
→
N
~
i
=
1
-
β
2
→
N
~
L
β
2
→
N
~
i
,
i
=
1,2
,
…
,
L
-
1
;
(c)
calculate the residual error after adding the new hidden nodeL
2
→
N
~:(32)
E
=
1
-
β
2
→
N
~
L
E
+
β
1
L
F
-
V
2
→
N
~
L
;Endwhile.The DOC-IELM-AEs algorithm inherits the advantages of incremental constructive feedforward networks model and deep learning algorithms on exactly capturing higher-level abstractions and characterizing the data representations. The implementation of autoencoders for the unsupervised pretraining of data exhibits super-duper performance on regression and classification problems. The improved method utilizes the OCI-ELM-AE as a base building layer to construct the whole deep architecture. As shown in Figure2, the data is mapped to OCI-ELM feature space; in each layer, OCI-ELM-AE output weights with respect to input data are the weight of the first layer; for the same reason, the output weights of OCI-ELM-AE, with respect to hidden layer output, are the layer weights of DOC-IELM-AEs. The detailed algorithm of DOC-IELM-AEs is shown in Algorithm 6.Figure 2
The model structure of DOC-IELM-AEs.


#### 5.2.1. Performance Comparison of Regression Problems Based on DOC-IELM-AEs

In this section, we mainly test the regression performance of the proposed OCI-ELM and DOC-IELM-AEs on three UCI real-world datasets, Parkinsons, California Housing, and CCS (Concrete Compressive Strength) data, and two large datasets, BlogFeedback and Online News Popularity data. The simulations are conducted in MATLAB 2013a environment running on Windows 7 machine with 128 GB of memory and Intel Xeon E5-2620V2 (2.1 GHz) processor.The regression performance comparisons of the proposed algorithms OCI-ELM and DOC-IELM-AEs with the baseline methods including SVM [35], single ELM, ML-ELM, AE-S-ELMs, DBN, ErrCor [36], and PC-ELM are shown in Table 5. The specific analyses on the results of regression capability and effectiveness are as follows:(1)
OCI-ELM compared with SVM, ELM, ErrCor, and PC-ELM: we perform the regression testing on the datasets described in Table4. The simulations are obtained by the average of 50 trails; we can observe from Table 4 that the testing accuracies of OCI-ELM on UCI datasets and large datasets are both better than SVM, ELM, ErrCor, and PC-ELM. For BlogFeedback dataset, the training accuracy of OCI-ELM is 91.76%, and those of SVM, ELM, ErrCor, and PC-ELM are 89.75%, 90.12%, 90.39%, and 90.54%, respectively. Meanwhile, OCI-ELM also obtains better testing accuracy of 91.82% than other algorithms. Although the OCI-ELM is an iterative learning algorithm, the compact neural network makes the convergence rate faster than PC-ELM and ErrCor, merely slower than SVM and ELM. Thus, the training time consumed on OCI-ELM learning is acceptable.
(2)
DOC-IELM-AEs compared with DBN, ML-ELM, and AE-S-ELMs: the testing accuracy on UCI datasets can show that the performance of DOC-IELM-AEs outperforms the OCI-ELM. With the aforementioned comparisons between OCI-ELM and the other algorithms (SVM, ELM, ErrCor, and PC-ELM), evidenced by the same token, the DOC-IELM-AEs can achieve better testing accuracy than SVM, ELM, ErrCor, and PC-ELM; this result can also be seen in Table5. For the large-scale datasets (BlogFeedback and Online News Popularity), the DOC-IELM-AEs obtained accuracies of 93.16%, 93.27% and 93.69%, 93.84% for training and testing with the network structures 281-1000-1000-2000-10 and 61-700-700-10000-26, respectively. The simulations in Table 5 show that DOC-IELM-AEs can produce better results than DBN, ML-ELM, and AE-S-ELMs. Furthermore, DOC-IELM-AEs enjoys the advantage over the DBN and ML-ELM on training speed. Thus, with the better regression performance, DOC-IELM-AEs would provide the state-of-the-art method for large-scale unstructured data problems.Table 4
Specification of 10 benchmark problems.
Datasets
Training sample
Testing sample
Attributes
Type
Parkinsons
4,780
5,120
26
Regression
California Housing
8,800
8,260
8
Regression
Concrete Compressive Strength
975
830
9
Regression
BlogFeedback
58,000
23,000
281
Regression
Online News Popularity
36,800
10,000
61
Regression
Delta Ailerons
5,080
4,600
6
Classification
Waveform II
3,300
2,800
40
Classification
MNIST
35,000
6,000
784
Classification
OCR Letters
40,000
12,000
128
Classification
NORB
20,000
5,000
2,048
ClassificationTable 5
The comparisons of training and testing on the regression cases.
Datasets
Algorithms
Training accuracy (%)
Training time (s)
Testing accuracy (%)
Testing deviation (%)
Parkinsons
SVM
95.42
282.26
95.43
0.01
ELM
95.75
240.68
95.73
0.29
ML-ELM
97.93
482.7
97.96
0.08
AE-S-ELMs
98.07
492.21
98.05
0.11
DBN
97.51
5013
97.53
0.04
ErrCor
96.25
344.82
96.19
0.32
PC-ELM
97.16
362.05
97.18
0.09
OCI-ELM
97.59
327.2
97.62
0.05
DOC-IELM-AEs
98.44
440.39
98.41
0.04
California Housing
SVM
96.56
327.17
96.56
0
ELM
96.79
289.24
96.72
0.16
ML-ELM
98.12
462.31
98.14
0.12
AE-S-ELMs
98.26
402.49
98.23
0.09
DBN
98.03
3989
98.04
0.08
ErrCor
97.01
401.66
97.11
0.25
PC-ELM
97.34
396.52
97.31
0.28
OCI-ELM
97.92
386.63
97.90
0.11
DOC-IELM-AEs
98.71
447.2
98.72
0.07
Concrete Compressive Strength
SVM
95.72
32.32
95.78
0.04
ELM
95.91
28.29
96.02
0.06
ML-ELM
97.26
53.4
97.33
0.18
AE-S-ELMs
97.49
50.66
97.41
0.12
DBN
96.64
320.04
96.58
0.05
ErrCor
96.32
47.45
96.35
0.25
PC-ELM
96.55
43.97
96.67
0.09
OCI-ELM
97.04
44.33
97.15
0.06
DOC-IELM-AEs
98.39
51.58
98.37
0.03
BlogFeedback
SVM
89.75
3906
89.82
0.03
ELM
90.12
2405
90.14
0.22
ML-ELM
91.86
5175
91.82
0.12
AE-S-ELMs
91.83
5247
91.79
0.13
DBN
90.51
19766
90.60
0.09
ErrCor
90.39
4889
90.44
0.09
PC-ELM
90.54
4308
90.58
0.12
OCI-ELM
91.76
4223
91.82
0.09
DOC-IELM-AEs
93.16
5271
93.27
0.07
Online News Popularity
SVM
91.75
311.83
91.72
0.04
ELM
91.68
131.34
91.69
0.34
ML-ELM
92.62
684.98
92.71
0.17
AE-S-ELMs
92.59
685.62
95.72
0.12
DBN
92.23
7062
92.26
0.05
ErrCor
91.61
541.14
91.77
0.29
PC-ELM
92.29
576.01
92.35
0.15
OCI-ELM
92.52
521.24
92.54
0.12
DOC-IELM-AEs
93.69
634.09
93.84
0.11


#### 5.2.2. Performance Comparison of Classification Problems Based on DOC-IELM-AEs

The classification performance comparisons of the proposed algorithms OCI-ELM and DOC-IELM-AEs with the baseline methods including SVM, single ELM, ML-ELM, AE-S-ELMs, DBN, ErrCor, and PC-ELM are shown in Table6. The specific comparisons are as follows:(1)
OCI-ELM compared with SVM, ELM, ErrCor, and PC-ELM: the simulation results are obtained by the average of 50 trails on datasets in Table4 (from Delta Ailerons to NORB data). For the BlogFeedback, the training and testing accuracies are 91.76% and 91.82%, respectively, listed in Table 6; we can see that OCI-ELM achieves better classification accuracy than SVM, ELM, ErrCor, and PC-ELM. And the speed of learning is faster than other improved ELM algorithms, notwithstanding behind the SVM and single ELM due to the process of iteration learning.
(2)
DOC-IELM-AEs compared with DBN, ML-ELM, and AE-S-ELMs: to test these anticipated effects, we used UCI datasets and large-scale datasets to acquire the results. From the experimental results, we can see that the classification accuracies of DOC-IELM-AEs are better than others obviously. Focusing on NORB, the network structure used by DOC-IELM-AEs is 2048-800-800-3000-5; the DOC-IELM-AEs obtained the best accuracies of 93.16%, 93.27% and 93.69%, 93.84% for training and testing, respectively, in all algorithms, including SVM, single ELM, ML-ELM, AE-S-ELMs, DBN, ErrCor, PC-ELM OCI-ELM, and DOC-IELM-AEs. Furthermore, the simulation results of other datasets also display the outstanding performance of DOC-IELM-AEs. Thus, with the better accuracy and faster speed of training, DOC-IELM-AEs can be applied in the vast majority of classification problems.Table 6
The comparisons of training and testing on the classification cases.
Datasets
Algorithms
Training accuracy (%)
Training time (s)
Testing accuracy (%)
Testing deviation (%)
Delta Ailerons
SVM
95.41
219.54
95.42
0
ELM
95.78
43.88
95.81
0.22
ML-ELM
97.32
317.83
97.34
0.19
AE-S-ELMs
97.75
279.32
97.71
0.27
DBN
98.34
4432
97.35
0.05
ErrCor
96.42
234.27
96.48
0.12
PC-ELM
96.56
262.69
96.61
0.08
OCI-ELM
96.73
249.89
96.69
0.07
DOC-IELM-AEs
98.89
350.17
98.90
0.05
Waveform II
SVM
95.45
247.11
95.44
0.01
ELM
95.71
107.14
95.73
0.16
ML-ELM
97.83
304.28
97.86
0.11
AE-S-ELMs
98.09
365.83
98.11
0.15
DBN
97.59
4101
97.52
0.04
ErrCor
96.54
261.44
96.58
0.06
PC-ELM
96.46
293.85
96.47
0.05
OCI-ELM
97.33
202.08
97.36
0.08
DOC-IELM-AEs
98.86
332.58
98.85
0.05
MNIST
SVM
95.01
2108
95.03
0.03
ELM
95.33
1114
95.32
0.31
ML-ELM
96.78
3793
96.82
0.17
AE-S-ELMs
96.93
3772
96.94
0.21
DBN
96.67
14117
96.72
0.03
ErrCor
96.17
4065
96.26
0.13
PC-ELM
96.21
3049
96.24
0.09
OCI-ELM
96.38
3092
96.39
0.09
DOC-IELM-AEs
97.89
3985
97.94
0.06
OCR Letters
SVM
88.27
770.24
88.41
0.03
ELM
89.02
265.01
89.03
0.22
ML-ELM
89.19
923.23
89.17
0.14
AE-S-ELMs
89.44
960.06
89.46
0.19
DBN
88.98
15212
89.02
0.04
ErrCor
88.81
1014
88.82
0.10
PC-ELM
89.17
979.62
89.16
0.05
OCI-ELM
89.36
992.08
89.39
0.08
DOC-IELM-AEs
90.85
1175
90.86
0.06
NORB
SVM
91.13
723.81
91.14
0.04
ELM
91.42
66.07
91.45
0.14
ML-ELM
92.67
1107
92.70
0.17
AE-S-ELMs
92.81
698.21
92.82
0.12
DBN
92.58
44502
92.59
0.03
ErrCor
91.77
1287
91.81
0.08
PC-ELM
91.80
896.63
91.83
0.06
OCI-ELM
92.61
839.48
92.65
0.05
DOC-IELM-AEs
94.53
1480
94.56
0.03


## 5.1. OCI-ELM Autoencoder

As an artificial neural network model, the autoencoder is frequently applied in the deep architecture approaches. Autoencoder is a kind of unsupervised neural network, where the input of network is equal to the output. Kasun et al. [29] proposed an autoencoder based on ELM (ELM-AE). According to their ELM-AE theory, the model of ELM-AE is composed of input layer, hidden layer, and output layer. In addition, the weights and biases of the hidden nodes are randomly generated via orthogonalization, and the input data is projected to a different or equal dimension space [30]; the expressions are as follows:(19)
h
=
σ
a
·
x
+
b
,
a
T
a
=
1
,
b
T
b
=
1
,
h
x
i
V
=
x
i
T
,
i
=
1,2
,
…
,
N
,where a
=
[
a
1
,
…
,
a
L
]
T are the weights generated orthogonally randomly and b
=
[
b
1
,
…
,
b
L
]
T are the biases generated orthogonally randomly between the input and hidden nodes. There are three calculation approaches to obtain the output weight β of ELM-AE: (1)
For sparse ELM-AE representations, output weightsβ can be calculated as follows:(20)
β
=
I
C
+
H
T
H
-
1
H
T
X
.
(2)
For compressed ELM-AE representations, output weightsβ can be calculated as follows:(21)
β
=
H
T
I
C
+
H
T
H
-
1
X
.
(3)
For equal dimension ELM-AE representations, output weightsβ can be calculated as follows:(22)
β
=
H
-
1
X
.In this section, we present the OCI-ELM, which is incorporated with Barron’s convex optimization learning method and Gram-Schmidt orthogonalization method into I-ELM to achieve the optimal least-squares solution as the training algorithm for an autoencoder instead of conventional autoencoders, which apply backpropagation algorithm (BP) for training to obtain the identity function and normal ELM for training the autoencoder. Because of the adoption of incremental algorithm, there is no need to set the number of hidden nodes according to the experience. With the initialization of the maximum valueL
m
a
x of hidden nodes the, number of hidden nodes can be increased by more than one node each time, until the stop criterions are met; for example, residual error E is equal to the expected learning accuracy ε or the number of hidden nodes L achieves L
m
a
x.As shown in Figure1, the model structure of OCI-ELM-AE can randomly control the number of the nodes without the computation accuracy. Given a training dataset ℵ
=
{
(
x
i
,
t
i
)
∣
x
i
∈
R
n
,
t
i
∈
R
m
,
i
=
1
,
…
,
N
}, where x
i
=
[
x
i
1
,
x
i
2
,
…
,
x
i
n
]
T and t
i
=
[
t
i
1
,
t
i
2
,
…
,
t
i
m
]
T, and given activation function σ
(
x
) and maximum number of hidden nodes in single layer L
m
a
x, the input data is reconstructed at the output layer through the following function:(23)
∑
i
=
1
l
β
i
σ
a
i
·
x
j
+
b
j
=
x
j
,
j
=
1,2
,
…
,
L
.The output weight can be obtained with the following:(24)
β
1
L
=
E
·
E
-
F
-
V
1
L
T
E
-
F
-
V
1
L
·
E
-
F
-
V
1
L
T
,
β
1
i
=
1
-
β
1
L
β
1
i
,
i
=
1,2
,
…
,
L
-
1
,where a
i
=
[
a
i
1
,
a
i
2
,
…
,
a
i
n
]
T is the input weight generated randomly and x
i
=
[
x
i
1
,
x
i
2
,
…
,
x
i
n
]
T are the input and output of the OCI-ELM-AE.Figure 1
Network diagram of OCI-ELM-AE.


## 5.2. Implementation of Stacked OCI-ELM Autoencoders in Deep Network

In 2006, Hinton et al. [31] presented the concept of deep learning to solve the problems of unsupervised data. Deep belief nets (DBNs) are probabilistic generative models which are first trained only with unlabeled data and then fine-tuned in a supervised mode. And then, another kind of deep network based on Restricted Boltzmann Machine (RBM) [32], deep Boltzmann machine (DBM) [33], was introduced by Salakhutdinov and Larochelle. The base building block of DBN is RBM. The ML-ELM was presented by Kasun et al. in 2013 [29]. There is no difference between the ML-ELM and the other deep learning models; ML-ELM performs layerwise unsupervised learning to train the parameters with the hidden layer weights which are initialized with ELM-AE, and the ML-ELM does not need to be fine-tuned. The AE-S-ELMs was proposed by Zhou et al. in 2014 [34]. The network consists of multiple ELMs with a small number of hidden nodes in each layer to substitute a single ELM with a large number of hidden nodes, and it implements ELM autoencoder in each iteration of S-ELMs algorithm to further improve the testing accuracy, especially for the unstructured large data without properly selected features.Algorithm 6 (deep network based on stacked orthogonal convex incremental ELM autoencoders (DOC-IELM-AEs)).
Given a training datasetℵ
=
{
(
x
i
,
t
i
)
∣
x
i
∈
R
n
,
t
i
∈
R
m
,
i
=
1,2
,
…
,
N
}, where x
i
=
[
x
i
1
,
x
i
2
,
…
,
x
i
n
]
T and t
i
=
[
t
i
1
,
t
i
n
,
…
,
t
i
m
], and given activation function σ
(
x
), maximum number of hidden nodes in single layer L
m
a
x, maximum number of iterations K
m
a
x, and expected learning accuracy ε, one has the following.Step 1 (initialization). Let the number of initial hidden nodes L
1
→
N
~
=
0, the number of iterations K
1
→
N
~
=
0, and residual error E
=
t, where t
=
[
t
1
,
t
-
2
,
…
,
t
N
]
T.Step 2 (orthogonal convex I-ELM autoencoder on layer 1). This step consists of two steps as follows.Orthogonalization Step. In this step, the following is carried out:(
a
)
Increase the number of hidden nodesL
1 and K
1 by one, respectively: L
1
=
L
1
+
1 and  K
1
=
K
1
+
1.
(
b
)
Randomly assign hidden node parameters(
a
L
,
b
L
) for new hidden node L and calculate the output g
k,(25)
v
k
=
g
k
-
v
1
,
g
k
v
1
,
v
1
v
1
-
v
2
,
g
k
v
2
,
v
2
v
2
-
⋯
-
v
k
-
1
,
g
k
v
k
-
1
,
v
k
-
1
v
k
-
1
,
and the hidden layer output matrix,(26)
V
=
V
1
σ
a
1
·
x
1
+
b
1
⋯
V
m
σ
a
m
·
x
1
+
b
m
⋮
⋯
⋮
V
1
σ
a
1
·
x
n
+
b
1
⋯
V
m
σ
a
m
·
x
n
+
b
m
n
×
m
.
Learning Step. While L
1
<
L
m
a
x, E
>
ε,(
c
)
calculate the output weightβ
1
(
L
) for the newly added hidden node: (27)
β
1
L
=
E
·
E
-
F
-
V
1
L
T
E
-
F
-
V
1
L
·
E
-
F
-
V
1
L
T
;
(
d
)
recalculate the output weight vectors of all existing hidden nodes ifL
1
>
1: (28)
β
1
i
=
1
-
β
1
L
β
1
i
,
i
=
1,2
,
…
,
L
-
1
;
(
e
)
calculate the residual error after adding the new hidden nodeL
1:(29)
E
=
1
-
β
1
L
E
+
β
1
L
F
-
V
1
L
.
Step 3 (orthogonal convex I-ELM autoencoder on layer 2
→
N
~). This step is carried out as follows.Learning Step. While L
2
→
N
~
<
L
m
a
x, E
>
ε,(a)
calculate the output weightβ
1
(
L
) for the newly added hidden node with the hidden layer output matrix V:(30)
β
2
→
N
~
L
=
E
·
E
-
F
-
V
2
→
N
~
L
T
E
-
F
-
V
2
→
N
~
L
·
E
-
F
-
V
2
→
N
~
L
T
;
(b)
recalculate the output weight vectors of all existing hidden nodes ifL
2
→
N
~
>
1:(31)
β
2
→
N
~
i
=
1
-
β
2
→
N
~
L
β
2
→
N
~
i
,
i
=
1,2
,
…
,
L
-
1
;
(c)
calculate the residual error after adding the new hidden nodeL
2
→
N
~:(32)
E
=
1
-
β
2
→
N
~
L
E
+
β
1
L
F
-
V
2
→
N
~
L
;Endwhile.The DOC-IELM-AEs algorithm inherits the advantages of incremental constructive feedforward networks model and deep learning algorithms on exactly capturing higher-level abstractions and characterizing the data representations. The implementation of autoencoders for the unsupervised pretraining of data exhibits super-duper performance on regression and classification problems. The improved method utilizes the OCI-ELM-AE as a base building layer to construct the whole deep architecture. As shown in Figure2, the data is mapped to OCI-ELM feature space; in each layer, OCI-ELM-AE output weights with respect to input data are the weight of the first layer; for the same reason, the output weights of OCI-ELM-AE, with respect to hidden layer output, are the layer weights of DOC-IELM-AEs. The detailed algorithm of DOC-IELM-AEs is shown in Algorithm 6.Figure 2
The model structure of DOC-IELM-AEs.


### 5.2.1. Performance Comparison of Regression Problems Based on DOC-IELM-AEs

In this section, we mainly test the regression performance of the proposed OCI-ELM and DOC-IELM-AEs on three UCI real-world datasets, Parkinsons, California Housing, and CCS (Concrete Compressive Strength) data, and two large datasets, BlogFeedback and Online News Popularity data. The simulations are conducted in MATLAB 2013a environment running on Windows 7 machine with 128 GB of memory and Intel Xeon E5-2620V2 (2.1 GHz) processor.The regression performance comparisons of the proposed algorithms OCI-ELM and DOC-IELM-AEs with the baseline methods including SVM [35], single ELM, ML-ELM, AE-S-ELMs, DBN, ErrCor [36], and PC-ELM are shown in Table 5. The specific analyses on the results of regression capability and effectiveness are as follows:(1)
OCI-ELM compared with SVM, ELM, ErrCor, and PC-ELM: we perform the regression testing on the datasets described in Table4. The simulations are obtained by the average of 50 trails; we can observe from Table 4 that the testing accuracies of OCI-ELM on UCI datasets and large datasets are both better than SVM, ELM, ErrCor, and PC-ELM. For BlogFeedback dataset, the training accuracy of OCI-ELM is 91.76%, and those of SVM, ELM, ErrCor, and PC-ELM are 89.75%, 90.12%, 90.39%, and 90.54%, respectively. Meanwhile, OCI-ELM also obtains better testing accuracy of 91.82% than other algorithms. Although the OCI-ELM is an iterative learning algorithm, the compact neural network makes the convergence rate faster than PC-ELM and ErrCor, merely slower than SVM and ELM. Thus, the training time consumed on OCI-ELM learning is acceptable.
(2)
DOC-IELM-AEs compared with DBN, ML-ELM, and AE-S-ELMs: the testing accuracy on UCI datasets can show that the performance of DOC-IELM-AEs outperforms the OCI-ELM. With the aforementioned comparisons between OCI-ELM and the other algorithms (SVM, ELM, ErrCor, and PC-ELM), evidenced by the same token, the DOC-IELM-AEs can achieve better testing accuracy than SVM, ELM, ErrCor, and PC-ELM; this result can also be seen in Table5. For the large-scale datasets (BlogFeedback and Online News Popularity), the DOC-IELM-AEs obtained accuracies of 93.16%, 93.27% and 93.69%, 93.84% for training and testing with the network structures 281-1000-1000-2000-10 and 61-700-700-10000-26, respectively. The simulations in Table 5 show that DOC-IELM-AEs can produce better results than DBN, ML-ELM, and AE-S-ELMs. Furthermore, DOC-IELM-AEs enjoys the advantage over the DBN and ML-ELM on training speed. Thus, with the better regression performance, DOC-IELM-AEs would provide the state-of-the-art method for large-scale unstructured data problems.Table 4
Specification of 10 benchmark problems.
Datasets
Training sample
Testing sample
Attributes
Type
Parkinsons
4,780
5,120
26
Regression
California Housing
8,800
8,260
8
Regression
Concrete Compressive Strength
975
830
9
Regression
BlogFeedback
58,000
23,000
281
Regression
Online News Popularity
36,800
10,000
61
Regression
Delta Ailerons
5,080
4,600
6
Classification
Waveform II
3,300
2,800
40
Classification
MNIST
35,000
6,000
784
Classification
OCR Letters
40,000
12,000
128
Classification
NORB
20,000
5,000
2,048
ClassificationTable 5
The comparisons of training and testing on the regression cases.
Datasets
Algorithms
Training accuracy (%)
Training time (s)
Testing accuracy (%)
Testing deviation (%)
Parkinsons
SVM
95.42
282.26
95.43
0.01
ELM
95.75
240.68
95.73
0.29
ML-ELM
97.93
482.7
97.96
0.08
AE-S-ELMs
98.07
492.21
98.05
0.11
DBN
97.51
5013
97.53
0.04
ErrCor
96.25
344.82
96.19
0.32
PC-ELM
97.16
362.05
97.18
0.09
OCI-ELM
97.59
327.2
97.62
0.05
DOC-IELM-AEs
98.44
440.39
98.41
0.04
California Housing
SVM
96.56
327.17
96.56
0
ELM
96.79
289.24
96.72
0.16
ML-ELM
98.12
462.31
98.14
0.12
AE-S-ELMs
98.26
402.49
98.23
0.09
DBN
98.03
3989
98.04
0.08
ErrCor
97.01
401.66
97.11
0.25
PC-ELM
97.34
396.52
97.31
0.28
OCI-ELM
97.92
386.63
97.90
0.11
DOC-IELM-AEs
98.71
447.2
98.72
0.07
Concrete Compressive Strength
SVM
95.72
32.32
95.78
0.04
ELM
95.91
28.29
96.02
0.06
ML-ELM
97.26
53.4
97.33
0.18
AE-S-ELMs
97.49
50.66
97.41
0.12
DBN
96.64
320.04
96.58
0.05
ErrCor
96.32
47.45
96.35
0.25
PC-ELM
96.55
43.97
96.67
0.09
OCI-ELM
97.04
44.33
97.15
0.06
DOC-IELM-AEs
98.39
51.58
98.37
0.03
BlogFeedback
SVM
89.75
3906
89.82
0.03
ELM
90.12
2405
90.14
0.22
ML-ELM
91.86
5175
91.82
0.12
AE-S-ELMs
91.83
5247
91.79
0.13
DBN
90.51
19766
90.60
0.09
ErrCor
90.39
4889
90.44
0.09
PC-ELM
90.54
4308
90.58
0.12
OCI-ELM
91.76
4223
91.82
0.09
DOC-IELM-AEs
93.16
5271
93.27
0.07
Online News Popularity
SVM
91.75
311.83
91.72
0.04
ELM
91.68
131.34
91.69
0.34
ML-ELM
92.62
684.98
92.71
0.17
AE-S-ELMs
92.59
685.62
95.72
0.12
DBN
92.23
7062
92.26
0.05
ErrCor
91.61
541.14
91.77
0.29
PC-ELM
92.29
576.01
92.35
0.15
OCI-ELM
92.52
521.24
92.54
0.12
DOC-IELM-AEs
93.69
634.09
93.84
0.11


### 5.2.2. Performance Comparison of Classification Problems Based on DOC-IELM-AEs

The classification performance comparisons of the proposed algorithms OCI-ELM and DOC-IELM-AEs with the baseline methods including SVM, single ELM, ML-ELM, AE-S-ELMs, DBN, ErrCor, and PC-ELM are shown in Table6. The specific comparisons are as follows:(1)
OCI-ELM compared with SVM, ELM, ErrCor, and PC-ELM: the simulation results are obtained by the average of 50 trails on datasets in Table4 (from Delta Ailerons to NORB data). For the BlogFeedback, the training and testing accuracies are 91.76% and 91.82%, respectively, listed in Table 6; we can see that OCI-ELM achieves better classification accuracy than SVM, ELM, ErrCor, and PC-ELM. And the speed of learning is faster than other improved ELM algorithms, notwithstanding behind the SVM and single ELM due to the process of iteration learning.
(2)
DOC-IELM-AEs compared with DBN, ML-ELM, and AE-S-ELMs: to test these anticipated effects, we used UCI datasets and large-scale datasets to acquire the results. From the experimental results, we can see that the classification accuracies of DOC-IELM-AEs are better than others obviously. Focusing on NORB, the network structure used by DOC-IELM-AEs is 2048-800-800-3000-5; the DOC-IELM-AEs obtained the best accuracies of 93.16%, 93.27% and 93.69%, 93.84% for training and testing, respectively, in all algorithms, including SVM, single ELM, ML-ELM, AE-S-ELMs, DBN, ErrCor, PC-ELM OCI-ELM, and DOC-IELM-AEs. Furthermore, the simulation results of other datasets also display the outstanding performance of DOC-IELM-AEs. Thus, with the better accuracy and faster speed of training, DOC-IELM-AEs can be applied in the vast majority of classification problems.Table 6
The comparisons of training and testing on the classification cases.
Datasets
Algorithms
Training accuracy (%)
Training time (s)
Testing accuracy (%)
Testing deviation (%)
Delta Ailerons
SVM
95.41
219.54
95.42
0
ELM
95.78
43.88
95.81
0.22
ML-ELM
97.32
317.83
97.34
0.19
AE-S-ELMs
97.75
279.32
97.71
0.27
DBN
98.34
4432
97.35
0.05
ErrCor
96.42
234.27
96.48
0.12
PC-ELM
96.56
262.69
96.61
0.08
OCI-ELM
96.73
249.89
96.69
0.07
DOC-IELM-AEs
98.89
350.17
98.90
0.05
Waveform II
SVM
95.45
247.11
95.44
0.01
ELM
95.71
107.14
95.73
0.16
ML-ELM
97.83
304.28
97.86
0.11
AE-S-ELMs
98.09
365.83
98.11
0.15
DBN
97.59
4101
97.52
0.04
ErrCor
96.54
261.44
96.58
0.06
PC-ELM
96.46
293.85
96.47
0.05
OCI-ELM
97.33
202.08
97.36
0.08
DOC-IELM-AEs
98.86
332.58
98.85
0.05
MNIST
SVM
95.01
2108
95.03
0.03
ELM
95.33
1114
95.32
0.31
ML-ELM
96.78
3793
96.82
0.17
AE-S-ELMs
96.93
3772
96.94
0.21
DBN
96.67
14117
96.72
0.03
ErrCor
96.17
4065
96.26
0.13
PC-ELM
96.21
3049
96.24
0.09
OCI-ELM
96.38
3092
96.39
0.09
DOC-IELM-AEs
97.89
3985
97.94
0.06
OCR Letters
SVM
88.27
770.24
88.41
0.03
ELM
89.02
265.01
89.03
0.22
ML-ELM
89.19
923.23
89.17
0.14
AE-S-ELMs
89.44
960.06
89.46
0.19
DBN
88.98
15212
89.02
0.04
ErrCor
88.81
1014
88.82
0.10
PC-ELM
89.17
979.62
89.16
0.05
OCI-ELM
89.36
992.08
89.39
0.08
DOC-IELM-AEs
90.85
1175
90.86
0.06
NORB
SVM
91.13
723.81
91.14
0.04
ELM
91.42
66.07
91.45
0.14
ML-ELM
92.67
1107
92.70
0.17
AE-S-ELMs
92.81
698.21
92.82
0.12
DBN
92.58
44502
92.59
0.03
ErrCor
91.77
1287
91.81
0.08
PC-ELM
91.80
896.63
91.83
0.06
OCI-ELM
92.61
839.48
92.65
0.05
DOC-IELM-AEs
94.53
1480
94.56
0.03


## 5.2.1. Performance Comparison of Regression Problems Based on DOC-IELM-AEs

In this section, we mainly test the regression performance of the proposed OCI-ELM and DOC-IELM-AEs on three UCI real-world datasets, Parkinsons, California Housing, and CCS (Concrete Compressive Strength) data, and two large datasets, BlogFeedback and Online News Popularity data. The simulations are conducted in MATLAB 2013a environment running on Windows 7 machine with 128 GB of memory and Intel Xeon E5-2620V2 (2.1 GHz) processor.The regression performance comparisons of the proposed algorithms OCI-ELM and DOC-IELM-AEs with the baseline methods including SVM [35], single ELM, ML-ELM, AE-S-ELMs, DBN, ErrCor [36], and PC-ELM are shown in Table 5. The specific analyses on the results of regression capability and effectiveness are as follows:(1)
OCI-ELM compared with SVM, ELM, ErrCor, and PC-ELM: we perform the regression testing on the datasets described in Table4. The simulations are obtained by the average of 50 trails; we can observe from Table 4 that the testing accuracies of OCI-ELM on UCI datasets and large datasets are both better than SVM, ELM, ErrCor, and PC-ELM. For BlogFeedback dataset, the training accuracy of OCI-ELM is 91.76%, and those of SVM, ELM, ErrCor, and PC-ELM are 89.75%, 90.12%, 90.39%, and 90.54%, respectively. Meanwhile, OCI-ELM also obtains better testing accuracy of 91.82% than other algorithms. Although the OCI-ELM is an iterative learning algorithm, the compact neural network makes the convergence rate faster than PC-ELM and ErrCor, merely slower than SVM and ELM. Thus, the training time consumed on OCI-ELM learning is acceptable.
(2)
DOC-IELM-AEs compared with DBN, ML-ELM, and AE-S-ELMs: the testing accuracy on UCI datasets can show that the performance of DOC-IELM-AEs outperforms the OCI-ELM. With the aforementioned comparisons between OCI-ELM and the other algorithms (SVM, ELM, ErrCor, and PC-ELM), evidenced by the same token, the DOC-IELM-AEs can achieve better testing accuracy than SVM, ELM, ErrCor, and PC-ELM; this result can also be seen in Table5. For the large-scale datasets (BlogFeedback and Online News Popularity), the DOC-IELM-AEs obtained accuracies of 93.16%, 93.27% and 93.69%, 93.84% for training and testing with the network structures 281-1000-1000-2000-10 and 61-700-700-10000-26, respectively. The simulations in Table 5 show that DOC-IELM-AEs can produce better results than DBN, ML-ELM, and AE-S-ELMs. Furthermore, DOC-IELM-AEs enjoys the advantage over the DBN and ML-ELM on training speed. Thus, with the better regression performance, DOC-IELM-AEs would provide the state-of-the-art method for large-scale unstructured data problems.Table 4
Specification of 10 benchmark problems.
Datasets
Training sample
Testing sample
Attributes
Type
Parkinsons
4,780
5,120
26
Regression
California Housing
8,800
8,260
8
Regression
Concrete Compressive Strength
975
830
9
Regression
BlogFeedback
58,000
23,000
281
Regression
Online News Popularity
36,800
10,000
61
Regression
Delta Ailerons
5,080
4,600
6
Classification
Waveform II
3,300
2,800
40
Classification
MNIST
35,000
6,000
784
Classification
OCR Letters
40,000
12,000
128
Classification
NORB
20,000
5,000
2,048
ClassificationTable 5
The comparisons of training and testing on the regression cases.
Datasets
Algorithms
Training accuracy (%)
Training time (s)
Testing accuracy (%)
Testing deviation (%)
Parkinsons
SVM
95.42
282.26
95.43
0.01
ELM
95.75
240.68
95.73
0.29
ML-ELM
97.93
482.7
97.96
0.08
AE-S-ELMs
98.07
492.21
98.05
0.11
DBN
97.51
5013
97.53
0.04
ErrCor
96.25
344.82
96.19
0.32
PC-ELM
97.16
362.05
97.18
0.09
OCI-ELM
97.59
327.2
97.62
0.05
DOC-IELM-AEs
98.44
440.39
98.41
0.04
California Housing
SVM
96.56
327.17
96.56
0
ELM
96.79
289.24
96.72
0.16
ML-ELM
98.12
462.31
98.14
0.12
AE-S-ELMs
98.26
402.49
98.23
0.09
DBN
98.03
3989
98.04
0.08
ErrCor
97.01
401.66
97.11
0.25
PC-ELM
97.34
396.52
97.31
0.28
OCI-ELM
97.92
386.63
97.90
0.11
DOC-IELM-AEs
98.71
447.2
98.72
0.07
Concrete Compressive Strength
SVM
95.72
32.32
95.78
0.04
ELM
95.91
28.29
96.02
0.06
ML-ELM
97.26
53.4
97.33
0.18
AE-S-ELMs
97.49
50.66
97.41
0.12
DBN
96.64
320.04
96.58
0.05
ErrCor
96.32
47.45
96.35
0.25
PC-ELM
96.55
43.97
96.67
0.09
OCI-ELM
97.04
44.33
97.15
0.06
DOC-IELM-AEs
98.39
51.58
98.37
0.03
BlogFeedback
SVM
89.75
3906
89.82
0.03
ELM
90.12
2405
90.14
0.22
ML-ELM
91.86
5175
91.82
0.12
AE-S-ELMs
91.83
5247
91.79
0.13
DBN
90.51
19766
90.60
0.09
ErrCor
90.39
4889
90.44
0.09
PC-ELM
90.54
4308
90.58
0.12
OCI-ELM
91.76
4223
91.82
0.09
DOC-IELM-AEs
93.16
5271
93.27
0.07
Online News Popularity
SVM
91.75
311.83
91.72
0.04
ELM
91.68
131.34
91.69
0.34
ML-ELM
92.62
684.98
92.71
0.17
AE-S-ELMs
92.59
685.62
95.72
0.12
DBN
92.23
7062
92.26
0.05
ErrCor
91.61
541.14
91.77
0.29
PC-ELM
92.29
576.01
92.35
0.15
OCI-ELM
92.52
521.24
92.54
0.12
DOC-IELM-AEs
93.69
634.09
93.84
0.11


## 5.2.2. Performance Comparison of Classification Problems Based on DOC-IELM-AEs

The classification performance comparisons of the proposed algorithms OCI-ELM and DOC-IELM-AEs with the baseline methods including SVM, single ELM, ML-ELM, AE-S-ELMs, DBN, ErrCor, and PC-ELM are shown in Table6. The specific comparisons are as follows:(1)
OCI-ELM compared with SVM, ELM, ErrCor, and PC-ELM: the simulation results are obtained by the average of 50 trails on datasets in Table4 (from Delta Ailerons to NORB data). For the BlogFeedback, the training and testing accuracies are 91.76% and 91.82%, respectively, listed in Table 6; we can see that OCI-ELM achieves better classification accuracy than SVM, ELM, ErrCor, and PC-ELM. And the speed of learning is faster than other improved ELM algorithms, notwithstanding behind the SVM and single ELM due to the process of iteration learning.
(2)
DOC-IELM-AEs compared with DBN, ML-ELM, and AE-S-ELMs: to test these anticipated effects, we used UCI datasets and large-scale datasets to acquire the results. From the experimental results, we can see that the classification accuracies of DOC-IELM-AEs are better than others obviously. Focusing on NORB, the network structure used by DOC-IELM-AEs is 2048-800-800-3000-5; the DOC-IELM-AEs obtained the best accuracies of 93.16%, 93.27% and 93.69%, 93.84% for training and testing, respectively, in all algorithms, including SVM, single ELM, ML-ELM, AE-S-ELMs, DBN, ErrCor, PC-ELM OCI-ELM, and DOC-IELM-AEs. Furthermore, the simulation results of other datasets also display the outstanding performance of DOC-IELM-AEs. Thus, with the better accuracy and faster speed of training, DOC-IELM-AEs can be applied in the vast majority of classification problems.Table 6
The comparisons of training and testing on the classification cases.
Datasets
Algorithms
Training accuracy (%)
Training time (s)
Testing accuracy (%)
Testing deviation (%)
Delta Ailerons
SVM
95.41
219.54
95.42
0
ELM
95.78
43.88
95.81
0.22
ML-ELM
97.32
317.83
97.34
0.19
AE-S-ELMs
97.75
279.32
97.71
0.27
DBN
98.34
4432
97.35
0.05
ErrCor
96.42
234.27
96.48
0.12
PC-ELM
96.56
262.69
96.61
0.08
OCI-ELM
96.73
249.89
96.69
0.07
DOC-IELM-AEs
98.89
350.17
98.90
0.05
Waveform II
SVM
95.45
247.11
95.44
0.01
ELM
95.71
107.14
95.73
0.16
ML-ELM
97.83
304.28
97.86
0.11
AE-S-ELMs
98.09
365.83
98.11
0.15
DBN
97.59
4101
97.52
0.04
ErrCor
96.54
261.44
96.58
0.06
PC-ELM
96.46
293.85
96.47
0.05
OCI-ELM
97.33
202.08
97.36
0.08
DOC-IELM-AEs
98.86
332.58
98.85
0.05
MNIST
SVM
95.01
2108
95.03
0.03
ELM
95.33
1114
95.32
0.31
ML-ELM
96.78
3793
96.82
0.17
AE-S-ELMs
96.93
3772
96.94
0.21
DBN
96.67
14117
96.72
0.03
ErrCor
96.17
4065
96.26
0.13
PC-ELM
96.21
3049
96.24
0.09
OCI-ELM
96.38
3092
96.39
0.09
DOC-IELM-AEs
97.89
3985
97.94
0.06
OCR Letters
SVM
88.27
770.24
88.41
0.03
ELM
89.02
265.01
89.03
0.22
ML-ELM
89.19
923.23
89.17
0.14
AE-S-ELMs
89.44
960.06
89.46
0.19
DBN
88.98
15212
89.02
0.04
ErrCor
88.81
1014
88.82
0.10
PC-ELM
89.17
979.62
89.16
0.05
OCI-ELM
89.36
992.08
89.39
0.08
DOC-IELM-AEs
90.85
1175
90.86
0.06
NORB
SVM
91.13
723.81
91.14
0.04
ELM
91.42
66.07
91.45
0.14
ML-ELM
92.67
1107
92.70
0.17
AE-S-ELMs
92.81
698.21
92.82
0.12
DBN
92.58
44502
92.59
0.03
ErrCor
91.77
1287
91.81
0.08
PC-ELM
91.80
896.63
91.83
0.06
OCI-ELM
92.61
839.48
92.65
0.05
DOC-IELM-AEs
94.53
1480
94.56
0.03


## 6. Case Study on Elongation Prediction of Strips

In this section, all of the experimental results for the elongation of strips prediction are presented. The annealing treatment is considered the most important process to cold rolled strips. In this process, the cold working hardening and internal stress of strips can be eliminated; the hardness of strips can be reduced; moreover, the ability of plastic deformation, stamping, and mechanical technique can be improved. Figure3 shows the process of continuous annealing. In the furnace, the strips will pass five temperature sections, that is, preheating section (PHS), heating section (HS), slow cooling section (SS), rapid cooling section (RCS), and equalising section (ES), and three tension sections, that is, SS tension section, RCS tension section, and HS tension section. Therefore, the strips will extend or shorten with the changes of temperature and tension. Meanwhile, the surface friction coefficient and the rotational speed of the tension rolls also affect the elongation of strips, rendering the weld position unable to be tracked accurately, having a great influence on the rate of finished product and the safety of air-knife. Thus, the proposed method DOC-IELM-AEs is applied in the annealing of strips process to obtain the position information of welds. The annealing process has 12 continuous process measurements and 10 manipulated variables according to the experience and mechanism analysis.Figure 3
General scheme of strip steel in annealing process.We collect the historical records in the last 16 months which can affect the position of the welding seam, including the temperature data of 5 sections, the tension data of 3 sections, and the speed data of 11 sections. We use data of 10 months for training and the following data of 6 months for testing. The comparison results of elongation of strips prediction are shown in Figure4. From the figures, we can see that the prediction results in the 6 months obtained based on 6 algorithms can all approximate the measured values. Although there are only little differences in the experimental results, the prediction based on DOC-IELM-AEs consistently outperforms the other methods in the comparisons.Figure 4
The comparisons of prediction on 6-month strip-elongation data.For further investigation on the prediction capabilities of DOC-IELM-AEs, the performances of algorithms are evaluated in terms of four criteria, that is, the mean absolute percentage error (M
A
P
E), the mean square error (M
S
E), the relative root-mean-square error (r
R
M
S
E), and the absolute fraction of variance (R
2). During testing of DOC-IELM-AEs and other algorithms, they are also defined by using the following equations:(33)
M
A
P
E
=
1
n
∑
i
=
1
n
y
i
-
y
1
′
y
i
×
100
%
,
M
S
E
=
1
n
∑
i
=
1
n
y
i
-
y
i
′
2
,
r
R
M
S
E
=
n
∑
i
=
1
n
y
i
-
y
i
′
2
n
-
1
∑
i
=
1
n
y
i
2
,
R
2
=
1
-
∑
i
=
1
n
y
i
-
y
i
′
2
∑
i
=
1
n
y
i
′
2
,where y
i and y
i
′ are measured value and predicted value, respectively, and n is the number of testing data. The smaller M
A
P
E, M
S
E, and r
R
M
S
E and the larger R
2 are indicative of better generalization performance of algorithm.M
A
P
E in Figure 5(a) evaluates the effect on measured values, generated by the disparity between the elongation values of steel-strips and the predicted values. Meanwhile, M
S
E and r
R
M
S
E in Figures 5(b) and 5(c), respectively, also reflect the dispersion of models, where M
S
E are more sensitive to the large errors compared with r
R
M
S
E because the squared errors amplify the large errors further. R
2 in Figure 5(d) is the distance between errors and predicted values. R
2 closer to 1 means that the algorithms have better performances. By analyzing the comparisons, it has become apparent that the results for evaluation criteria based on arbitrary 6-month testing data show better generalization performances of DOC-IELM-AEs than other algorithms in experiments of comparison. Accordingly, there is important practical significance in the prediction of elongation of steel-strips using DOC-IELM-AEs.Figure 5
Different prediction errors of 5 models on 6-month data.
(a)
(b)
(c)
(d)In order to demonstrate the effectiveness of the algorithm proposed in practical engineering, we have selected the successive data of 12 months from the whole data (16-month data) to conduct the comparisons, and we obtained the prediction accuracies of every month and one year. The comparisons of prediction accuracy shown in Figure6 indicate that the performance of the algorithm proposed is the best overall, with mean accuracy of 96.795% in 12 months (all the year), compared with those of 92.49%, 92.71%, 94.47%, 94.62%, 94.43%, 93.18%, 93.22%, and 94.50% obtained from SVM, ELM, ML-ELM, AE-S-ELMs, DBN, ErrCor, PC-ELM, and OCI-ELM methods, respectively. DOC-IELM-AEs has the best accuracy among the nine methods, which indicates the predictive stability and performance of the method. Therefore, we can get the conclusion that DOC-IELM-AEs has the best prediction performance in the testing, and the algorithm proposed is a very effective method.Figure 6
The comparisons of prediction accuracy of annual statistical data.


## 7. Conclusions

In this paper, we proposed a stacked architecture with OCI-ELM algorithm based on deep representation learning and added the OCI-ELM autoencoder into each layer of OCI-ELM, called DOC-IELM-AEs. The experiment results have demonstrated strongly that DOC-IELM-AEs can be suitable for solving regression and classification problems; simulations showed that, (1) compared with CI-ELM, EI-ELM, ECI-ELM, PC-ELM, and OCI-ELM, DOC-IELM-AEs can achieve the best testing accuracy with the same network size, even less hidden nodes; meanwhile, the speed of learning is also faster than other algorithms. Moreover, DOC-IELM-AEs has better performance than OCI-ELM algorithm; (2) compared with SVM, ELM, ML-ELM, AE-S-ELMs, DBN, ErrCor, PC-ELM, and OCI-ELM, DOC-IELM-AEs can also obtain the best testing accuracy with consuming more time in a certain range for the large datasets; (3) compared with SVM, ELM, ML-ELM, AE-S-ELMs, DBN, ErrCor, PC-ELM, and OCI-ELM, the DOC-IELM-AEs applied in the case of strips-elongation prediction can enhance the performance of prediction; demonstrated with the production data, the prediction accuracy based on the algorithm we proposed outperforms other algorithms. For these reasons, the OCI-ELM and DOC-IELM-AEs can further be implemented in practical engineering and have the potential for solving more complicated big data problems with further study.



---

*Source: 1649486-2016-06-30.xml*