# Improved Extreme Learning Machine and Its Application in Image Quality Assessment

Authors: Li Mao; Lidong Zhang; Xingyang Liu; Chaofeng Li; Hong Yang
Journal: Mathematical Problems in Engineering
 (2014)

---

## Abstract

Extreme learning machine (ELM) is a new class of single-hidden layer feedforward neural network (SLFN), which is simple in theory and fast in implementation. Zong et al. propose a weighted extreme learning machine for learning data with imbalanced class distribution, which maintains the advantages from original ELM. However, the current reported ELM and its improved version are only based on the empirical risk minimization principle, which may suffer from overfitting. To solve the overfitting troubles, in this paper, we incorporate the structural risk minimization principle into the (weighted) ELM, and propose a modified (weighted) extreme learning machine (M-ELM and M-WELM). Experimental results show that our proposed M-WELM outperforms the current reported extreme learning machine algorithm in image quality assessment.


---

## Body

## 1. Introduction

Extreme learning machine (ELM) was proposed as a new class of single-hidden layer feedforward neural network by Huang et al. [1]. Its basic idea is to set a suitable number of nodes in the hidden layer before training and randomly assign the values for the input weights and offsets of the hidden layer in the implementation procedure. The algorithm completes the whole process at once and generates a unique optimal solution without the necessity of iterations. So it has the advantages of easy parameter selection and fast learning speed. Liang et al. [2] also proposed an online sequential extreme learning machine algorithm (OS-ELM) that can learn data one-by-one or chunk-by-chunk. Although OS-ELM provides better generalization performance, it excessively depends on experimental data. Lan et al. [3] presented an ensemble of online sequential extreme learning machine (EOS-ELM), which is a more stable integrated network structure consisting of multiple OS-ELM networks. Rong et al. [4] developed an OS-Fuzzy-ELM algorithm by combining TSK fuzzy inference system and ELM algorithm, which reduces the training time significantly. Feng et al. [5] presented an improved ELM algorithm based on error minimization. In [6], Zong et al. proposed a weighted ELM for dealing with data with imbalanced class distribution, which is able to be generalized to balanced data and maintains the advantages from original ELM. However all these algorithms only consider the empirical risk minimization principle, which can easily lead to overfitting [7].Support vector machine (SVM), proposed by Cortes and Vapnik [8], is actually also a single-hidden layer feedforward network. In [9–11], Suykens et al. proposed the least-squares support vector machine (LS-SVM), which transforms the linear inequality constraints into linear equality constraints in the support vector machine and, thus, converts solving the QP problem into solving linear equations. It reduces the difficulty of support vector machine learning a great deal of samples and also improves the efficiency. Both the SVM and LS-SVM are general algorithms based on guaranteed risk bounds of statistical learning theory, that is, the so-called structural risk minimization (SRM) principle, which improves their generalization ability.In this paper, to lower the overfitting phenomena of the extreme learning machine algorithms, we refer to the LS-SVM algorithm, draw the structural risk minimization principle into the ELM and WELM algorithms, and propose a modified ELM and WELM algorithm and call them as M-ELM and M-WELM. Our experimental results suggest the validity of our proposed M-ELM and M-WELM algorithm.The structure of this paper is organized as follows. The brief introduction to ELM and weighted ELM is done in Section2. In Section 3, the principles of our proposed M-ELM and M-WELM are described. Experimental results and performance assessment are presented in Section 4. In Section 5, the conclusion is presented.


## 2. Brief Introduction to Extreme Learning Machine

### 2.1. Extreme Learning Machine (ELM)

Extreme learning machine (ELM) proposed is a single-hidden layer feedforward networks (SLFNs) which randomly selected the input weights and analytically determines the output weights of SLFNs [1, 2, 12]. One key principle of the ELM is that one may randomly choose and fix the hidden node parameters. After the hidden nodes parameters are chosen randomly, SLFN becomes a linear system where the output weights of the network can be analytically determined using simple generalized inverse operation of the hidden layer output matrices [13].For an observation data set withN nodes in the hidden layer and the excitation function G, the extreme learning machine model can be expressed as
(1)
f
(
x
)
=
∑
i
=
1
N
β
i
G
(
a
i
,
b
i
,
x
i
)
=
β
·
h
(
x
)
,
where β
i is the output weight of the ith hidden layer node and the output neuron, a
i is the input weight of the input neuron and the ith hidden layer node, and b
i is the offset of the ith hidden layer node. Consider h
(
x
)
=
[
G
(
a
1
,
b
1
,
x
1
)
,
…
,
G
(
a
N
,
b
N
,
x
N
)
] denotes the output matrix of hidden layer. a
i and b
i are randomly selected before training and remain the same in the training procedure. The output weights β
i can be obtained by solving the least-squares solutions of the following linear equation:
(2)
min
⁡
∑
i
=
1
N
∥
β
i
·
h
(
x
i
)
-
y
i
∥
.
The least-squares solution to the equations is
(3)
β
=
H
+
Y
,
where H
+ is called the Moore-Penrose generalized inverse of the hidden layer output matrix H.


### 2.2. LS-SVM Regression

Assume that an input and output sample data set for regression analysis isT
=
{
(
x
1
,
y
1
)
,
…
,
(
x
l
,
y
l
)
}, where x
i
∈
R
n and y
∈
R, i
=
1
,
…
,
l. LS-SVM regression algorithm maps the data X into a high-dimensional feature space F through a nonlinear mapping ϕ and does linear regression in the space F. The regression estimation for the observation data set given above can be formulated as below, where ω and b
* are the regression factors,
(4)
f
(
x
)
=
(
ω
·
ϕ
(
x
)
)
+
b
*
.LS-SVM regression method is used to solve the weight vectorω and deviation b
*. Based on the structural risk minimization, the optimization model of the optimal regression function [9–11] can be established as
(5)
min
⁡
1
2
∥
ω
∥
2
+
1
2
γ
∑
i
+
1
l
ξ
i
2
s
.
t
.
(
subject
to
)
y
i
-
f
(
x
i
)
=
ξ
i
ξ
i
≥
0
,
i
=
1,2
,
…
,
l
,
where γ is the penalty constant, which is a compromise between complexity and fitting accuracy of regression model. Higher value γ means higher fitting degree. ξ
i is the slack variable. LS-SVM transforms the inequality constraints into equality constraints by defining loss functions different from those in the standard SVM. It constructs the following Lagrange function:
(6)
L
LS-SVM
=
1
2
∥
ω
∥
2
+
1
2
γ
∑
i
+
1
l
ξ
i
2
-
∑
i
+
1
l
α
i
[
y
i
-
f
(
x
i
)
-
ξ
i
]
,
where α
i is the Lagrange multiplier. According to KKT optimal conditions, the linear equations can be obtained as follows:
(7)
[
0
1
v
T
1
v
Ω
+
I
γ
]
[
b
*
α
]
=
[
0
y
]
,
where y
=
[
y
1
,
y
2
,
…
,
y
l
]
T, 1
v
=
[
1,1
,
…
,
1
]
T, 1
v
=
[
1,1
,
…
,
1
]
T, I is the l
×
l identity matrix, Ω is a square matrix, and Ω
i
,
j
=
K
(
x
i
,
x
j
) is the ith row and the jth column data element, where Ω
i
,
j
=
K
(
x
i
,
x
j
) is the kernel function that satisfies the Mercer condition.Solve the linear equations and get the nonlinear mapping equation as follows:(8)
f
(
x
)
=
∑
i
+
1
l
α
i
K
(
x
i
,
x
)
+
b
*
.


### 2.3. Weighted Extreme Learning Machine (WELM)

#### 2.3.1. Basic Theory

In [6], the authors proposed a weighted extreme learning machine for imbalance learning, which defined an N
×
N diagonal matrix W associated with every training sample x
i. Usually if training data x
i comes from a minority class (assumed to be positive class), the associated weight W
i
i will be set relatively larger than others. To maximize the marginal distance and to minimize the weighted cumulative error with respect to each sample, an optimization problem mathematically are written as
(9)
Minimize:
∥
H
β
-
T
∥
2
,
∥
β
∥
,
where T
=
[
t
1
,
…
,
t
N
]. More precisely,
(10)
Minimize:
L
P
ELM
=
1
2
∥
β
∥
2
+
1
2
C
W
∑
i
=
1
N
∥
ξ
i
∥
2
Subject
to:
h
(
x
i
)
β
=
t
i
T
-
ξ
i
T
,
i
=
1
,
…
,
N
,
where h
(
x
i
) is the feature mapping vector in the hidden layer with respect to x
i, β represents the output weight vector connecting the hidden layer and output layer, and C is the regularization parameter to represent the trade-off between the minimization of training errors and the maximization of the marginal distance. ξ
i, the training error of sample x
i, is caused by the difference of the desired output t
i and the actual output h
(
x
i
)
β.


#### 2.3.2. Weighting Schemes

The key issue of the WELM is to define an appropriate weight matrixW
=
diag
⁡
{
W
i
i
}, i
=
1
,
…
,
N, which determines what degree of rebalance users are seeking for and how much further the boundary is pushed towards the majority class [6]. In [6], two weighting schemes are proposed.The simple one is the weight value that can be automatically generated from the class information, which is in fact a special case of the cost sensitive learning:(11)
Weighting
Scheme
W
1
:
W
i
i
=
1
#
(
t
i
)
,
where #
(
t
i
) is the number of samples belonging to class t
i,  i
=
1
,
…
,
m.Another weighting scheme is the authors of [6] adopts the value of golden standard that represents the perfection in nature and minishes the balancing step into the ratio of 0.618 : 1 between minority classes and the majority classes, as shown in
(12)
Weighting
Scheme
W
2
{
W
i
i
=
0.618
#
(
t
i
)
if
t
i
>
AVG
(
t
i
)
W
i
i
=
1
#
(
t
i
)
if
t
i
≤
AVG
(
t
i
)
.Compared to weighting scheme W1, the boundary using weighting scheme W2 is pushed slightly backwards the minority class so that the misclassification cases in compromise on the majority side are sought of being alleviated, so we adopt weighting scheme W2 in the next experiments.


## 2.1. Extreme Learning Machine (ELM)

Extreme learning machine (ELM) proposed is a single-hidden layer feedforward networks (SLFNs) which randomly selected the input weights and analytically determines the output weights of SLFNs [1, 2, 12]. One key principle of the ELM is that one may randomly choose and fix the hidden node parameters. After the hidden nodes parameters are chosen randomly, SLFN becomes a linear system where the output weights of the network can be analytically determined using simple generalized inverse operation of the hidden layer output matrices [13].For an observation data set withN nodes in the hidden layer and the excitation function G, the extreme learning machine model can be expressed as
(1)
f
(
x
)
=
∑
i
=
1
N
β
i
G
(
a
i
,
b
i
,
x
i
)
=
β
·
h
(
x
)
,
where β
i is the output weight of the ith hidden layer node and the output neuron, a
i is the input weight of the input neuron and the ith hidden layer node, and b
i is the offset of the ith hidden layer node. Consider h
(
x
)
=
[
G
(
a
1
,
b
1
,
x
1
)
,
…
,
G
(
a
N
,
b
N
,
x
N
)
] denotes the output matrix of hidden layer. a
i and b
i are randomly selected before training and remain the same in the training procedure. The output weights β
i can be obtained by solving the least-squares solutions of the following linear equation:
(2)
min
⁡
∑
i
=
1
N
∥
β
i
·
h
(
x
i
)
-
y
i
∥
.
The least-squares solution to the equations is
(3)
β
=
H
+
Y
,
where H
+ is called the Moore-Penrose generalized inverse of the hidden layer output matrix H.


## 2.2. LS-SVM Regression

Assume that an input and output sample data set for regression analysis isT
=
{
(
x
1
,
y
1
)
,
…
,
(
x
l
,
y
l
)
}, where x
i
∈
R
n and y
∈
R, i
=
1
,
…
,
l. LS-SVM regression algorithm maps the data X into a high-dimensional feature space F through a nonlinear mapping ϕ and does linear regression in the space F. The regression estimation for the observation data set given above can be formulated as below, where ω and b
* are the regression factors,
(4)
f
(
x
)
=
(
ω
·
ϕ
(
x
)
)
+
b
*
.LS-SVM regression method is used to solve the weight vectorω and deviation b
*. Based on the structural risk minimization, the optimization model of the optimal regression function [9–11] can be established as
(5)
min
⁡
1
2
∥
ω
∥
2
+
1
2
γ
∑
i
+
1
l
ξ
i
2
s
.
t
.
(
subject
to
)
y
i
-
f
(
x
i
)
=
ξ
i
ξ
i
≥
0
,
i
=
1,2
,
…
,
l
,
where γ is the penalty constant, which is a compromise between complexity and fitting accuracy of regression model. Higher value γ means higher fitting degree. ξ
i is the slack variable. LS-SVM transforms the inequality constraints into equality constraints by defining loss functions different from those in the standard SVM. It constructs the following Lagrange function:
(6)
L
LS-SVM
=
1
2
∥
ω
∥
2
+
1
2
γ
∑
i
+
1
l
ξ
i
2
-
∑
i
+
1
l
α
i
[
y
i
-
f
(
x
i
)
-
ξ
i
]
,
where α
i is the Lagrange multiplier. According to KKT optimal conditions, the linear equations can be obtained as follows:
(7)
[
0
1
v
T
1
v
Ω
+
I
γ
]
[
b
*
α
]
=
[
0
y
]
,
where y
=
[
y
1
,
y
2
,
…
,
y
l
]
T, 1
v
=
[
1,1
,
…
,
1
]
T, 1
v
=
[
1,1
,
…
,
1
]
T, I is the l
×
l identity matrix, Ω is a square matrix, and Ω
i
,
j
=
K
(
x
i
,
x
j
) is the ith row and the jth column data element, where Ω
i
,
j
=
K
(
x
i
,
x
j
) is the kernel function that satisfies the Mercer condition.Solve the linear equations and get the nonlinear mapping equation as follows:(8)
f
(
x
)
=
∑
i
+
1
l
α
i
K
(
x
i
,
x
)
+
b
*
.


## 2.3. Weighted Extreme Learning Machine (WELM)

### 2.3.1. Basic Theory

In [6], the authors proposed a weighted extreme learning machine for imbalance learning, which defined an N
×
N diagonal matrix W associated with every training sample x
i. Usually if training data x
i comes from a minority class (assumed to be positive class), the associated weight W
i
i will be set relatively larger than others. To maximize the marginal distance and to minimize the weighted cumulative error with respect to each sample, an optimization problem mathematically are written as
(9)
Minimize:
∥
H
β
-
T
∥
2
,
∥
β
∥
,
where T
=
[
t
1
,
…
,
t
N
]. More precisely,
(10)
Minimize:
L
P
ELM
=
1
2
∥
β
∥
2
+
1
2
C
W
∑
i
=
1
N
∥
ξ
i
∥
2
Subject
to:
h
(
x
i
)
β
=
t
i
T
-
ξ
i
T
,
i
=
1
,
…
,
N
,
where h
(
x
i
) is the feature mapping vector in the hidden layer with respect to x
i, β represents the output weight vector connecting the hidden layer and output layer, and C is the regularization parameter to represent the trade-off between the minimization of training errors and the maximization of the marginal distance. ξ
i, the training error of sample x
i, is caused by the difference of the desired output t
i and the actual output h
(
x
i
)
β.


### 2.3.2. Weighting Schemes

The key issue of the WELM is to define an appropriate weight matrixW
=
diag
⁡
{
W
i
i
}, i
=
1
,
…
,
N, which determines what degree of rebalance users are seeking for and how much further the boundary is pushed towards the majority class [6]. In [6], two weighting schemes are proposed.The simple one is the weight value that can be automatically generated from the class information, which is in fact a special case of the cost sensitive learning:(11)
Weighting
Scheme
W
1
:
W
i
i
=
1
#
(
t
i
)
,
where #
(
t
i
) is the number of samples belonging to class t
i,  i
=
1
,
…
,
m.Another weighting scheme is the authors of [6] adopts the value of golden standard that represents the perfection in nature and minishes the balancing step into the ratio of 0.618 : 1 between minority classes and the majority classes, as shown in
(12)
Weighting
Scheme
W
2
{
W
i
i
=
0.618
#
(
t
i
)
if
t
i
>
AVG
(
t
i
)
W
i
i
=
1
#
(
t
i
)
if
t
i
≤
AVG
(
t
i
)
.Compared to weighting scheme W1, the boundary using weighting scheme W2 is pushed slightly backwards the minority class so that the misclassification cases in compromise on the majority side are sought of being alleviated, so we adopt weighting scheme W2 in the next experiments.


## 2.3.1. Basic Theory

In [6], the authors proposed a weighted extreme learning machine for imbalance learning, which defined an N
×
N diagonal matrix W associated with every training sample x
i. Usually if training data x
i comes from a minority class (assumed to be positive class), the associated weight W
i
i will be set relatively larger than others. To maximize the marginal distance and to minimize the weighted cumulative error with respect to each sample, an optimization problem mathematically are written as
(9)
Minimize:
∥
H
β
-
T
∥
2
,
∥
β
∥
,
where T
=
[
t
1
,
…
,
t
N
]. More precisely,
(10)
Minimize:
L
P
ELM
=
1
2
∥
β
∥
2
+
1
2
C
W
∑
i
=
1
N
∥
ξ
i
∥
2
Subject
to:
h
(
x
i
)
β
=
t
i
T
-
ξ
i
T
,
i
=
1
,
…
,
N
,
where h
(
x
i
) is the feature mapping vector in the hidden layer with respect to x
i, β represents the output weight vector connecting the hidden layer and output layer, and C is the regularization parameter to represent the trade-off between the minimization of training errors and the maximization of the marginal distance. ξ
i, the training error of sample x
i, is caused by the difference of the desired output t
i and the actual output h
(
x
i
)
β.


## 2.3.2. Weighting Schemes

The key issue of the WELM is to define an appropriate weight matrixW
=
diag
⁡
{
W
i
i
}, i
=
1
,
…
,
N, which determines what degree of rebalance users are seeking for and how much further the boundary is pushed towards the majority class [6]. In [6], two weighting schemes are proposed.The simple one is the weight value that can be automatically generated from the class information, which is in fact a special case of the cost sensitive learning:(11)
Weighting
Scheme
W
1
:
W
i
i
=
1
#
(
t
i
)
,
where #
(
t
i
) is the number of samples belonging to class t
i,  i
=
1
,
…
,
m.Another weighting scheme is the authors of [6] adopts the value of golden standard that represents the perfection in nature and minishes the balancing step into the ratio of 0.618 : 1 between minority classes and the majority classes, as shown in
(12)
Weighting
Scheme
W
2
{
W
i
i
=
0.618
#
(
t
i
)
if
t
i
>
AVG
(
t
i
)
W
i
i
=
1
#
(
t
i
)
if
t
i
≤
AVG
(
t
i
)
.Compared to weighting scheme W1, the boundary using weighting scheme W2 is pushed slightly backwards the minority class so that the misclassification cases in compromise on the majority side are sought of being alleviated, so we adopt weighting scheme W2 in the next experiments.


## 3. Modified Extreme Learning Machine Algorithm

The traditional extreme learning machines are based on the empirical risk minimization principle and the training error minimization principle, whose drawback is that it is likely to suffer from overfitting, which reduces the generalization capability consequently.According to the statistical theory, the actual risks include the empirical and structural risks, and a model with good generalization performance should be able to balance empirical and structural risks to obtain the best compromise. So we lead the structural risk minimization principle into the ELM algorithm and propose a modified weighted ELM and WELM model based on ELM and WELM, which we call it as M-ELM and M-WELM.Assume that an input and output sample data set for regression analysis isT
=
{
(
x
1
,
y
1
)
,
…
,
(
x
1
,
y
1
)
}, where x
i
∈
R
n andy
∈
R, i
=
1
,
…
,
l. We draw into the condition of the structural risk and adjust the proportion of the empirical and structural risks by ζ instead of the C in formula (10), and the optimization model of the optimal regression function can be established as follows:
(13)
min
⁡
1
2
∥
β
∥
2
+
1
2
ζ
∑
i
=
1
N
∥
δ
i
∥
2
for
M-ELM
or
min
⁡
1
2
∥
β
∥
2
+
1
2
ζ
W
∑
i
=
1
N
∥
δ
i
∥
2
for
M-WELM
Subject
to
{
y
i
-
β
i
h
(
x
i
)
=
δ
i
δ
i
≥
0
,
i
=
1
,
…
,
N
,
where δ
i
2, the sum of the square errors, represents the empirical risk and ∥
β
∥
2 represents the structural risk, according to the maximal margin principle in statistical theory [2]. According to formula (6), the formula above is the conditional extreme problem and can be transformed into the Lagrange equation as follows:
(14)
L
ELM
=
1
2
∥
β
∥
2
+
1
2
ζ
W
∑
i
=
1
N
∥
δ
i
∥
2
-
∑
i
=
1
N
α
i
[
y
i
-
β
h
(
x
i
)
-
δ
i
]
,
where the Lagrange multiplier α
i is the constant factor of sample x
i in the linear combination to form the final decision function. Further, by making the partial derivatives with respect to variables (
β
,
δ
i
,
α
i
) all equal to zero, the KKT optimality conditions are obtained:
(15)
∂
L
∂
β
=
0
⟹
β
=
∑
i
=
1
N
α
i
h
(
x
i
)
=
H
T
α
,
(16)
∂
L
∂
ξ
i
=
0
⟶
α
i
=
ζ
W
δ
i
,
i
=
1
,
…
,
N
,
(17)
∂
L
∂
α
i
=
0
⟹
y
i
-
β
i
h
(
x
i
)
-
δ
i
=
0
.The solution ofβ can be derived from (17) regarding left pseudoinverse. Usually, left pseudoinverse is more suitable since it is much easier to compute matrix inversion of size L
×
L, when L is much smaller than N:
(18)
β
=
(
1
ζ
+
H
T
W
H
)
-
1
H
T
W
T
.The same as formula (7) in Section 2.2, we can obtain the following linear equations:
(19)
[
0
1
v
T
1
v
Ω
+
I
ζ
]
[
0
α
]
=
[
0
y
]
,
where y
=
[
y
1
,
y
2
,
…
,
y
N
]
T; 1
v
=
[
1,1
,
…
,
1
]
T; α
=
[
α
1
,
α
2
,
…
,
α
N
]; I is the N
×
N identity matrix; Ω is a square matrix, and the ith row and the jth column data element is
(20)
Ω
i
,
j
=
K
ELM
(
x
i
,
x
j
)
=
h
(
x
i
)
·
h
(
x
j
)
=
[
G
(
a
1
,
b
1
,
x
i
)
,
…
,
G
(
a
N
,
b
N
,
x
i
)
]
·
[
G
(
a
1
,
b
1
,
x
i
)
,
…
,
G
(
a
N
,
b
N
,
x
i
)
]
T
,
where G is the excitation function.The sigmoid function is used in this paper as follows:(21)
G
(
a
,
b
,
x
)
=
1
(
1
+
exp
⁡
(
-
(
a
·
x
)
+
b
)
)
.Solve the linear equations and then get the following nonlinear mapping equation below that is derived from (8):
(22)
f
(
x
)
=
∑
i
=
1
N
α
i
K
ELM
(
x
i
,
x
)
.The whole steps of the M-ELM or M-WELM algorithm can be summarized as follows.Given a training setT, activation function G, and hidden node number N, consider the following.Step 1.
Transform formula (13) of conditional extreme problem into formula (9) of Lagrange equation.Step 2.
Calculateα
i using formulas (16) and (17).Step 3.
Substituteα
i into formula (15) and calculate the output weight β.As it can be seen, the M-WELM is able to be generalized to cost sensitive learning and can also deal with data with imbalanced class distribution as the WELM. On the other hand, its overfitting risk can be reduced by considering both the empirical and structural risks simultaneously.


## 4. Experiments and Performance Assessment

In this section, we present the performance comparison of proposed M-WELM and M-ELM and current reported ELM, OS-ELM [2], EOS-ELM [3], B-ELM [15], and C-ELM [14], and classifiers on benchmark prediction data sets first and then show the results for the image quality estimation problem. In the next whole experiments, the activation functions of ELM adopt sigmoid functions, and other parameters are gained by the cross validation method.


### 4.1. Test on the Benchmark Boston Housing Data Set

Boston Housing data, obtained from the UCI database, is a data set commonly used for measuring the performance of regression algorithm. It contains the information of 506 sets of commodity houses in Boston Housing, including 12 continuous characteristics, one discrete characteristic, and house prices [16]. The purpose of regression estimation is to predict the average house price by training part of the samples.In the experiments, the samples are randomly divided into two sample groups: random 70% of them for training and the remaining 30% for test. We repeat the random train-test procedure 100 times and calculate the mean square training and prediction error of every algorithm, and the experimental results of several algorithms are shown in Table1. We have adjusted the parameters for every algorithm so that every algorithm can get a pretty good result. The number of hidden neurons of M-WELM, M-ELM, W-ELM, OS-ELM, and EOS-ELM is set to 180, and the number of hidden neurons of ELM, B-ELM, and C-ELM is set to 65.Table 1
The experimental results of the several algorithms on the Boston Housing data set.
Algorithm
Training error
Prediction error
M-WELM
2.253
3.364
M-ELM
2.154
3.592
W-ELM [6]
2.391
3.624
ELM
2.723
4.090
OS-ELM [2]
2.538
3.912
EOS-ELM [3]
2.686
3.745
B-ELM [12]
2.930
3.999
C-ELM [13]
2.615
3.653It can be seen from Table1, for the Boston Housing data set from a real-world multi-input single-output system, that our proposed M-WELM algorithm shows the best prediction performance than other types of ELMs, and M-ELM ranks number 2, with both of which indicating the robustness of our proposed idea for modifying ELM algorithm.


### 4.2. Test on the LIVE IQA Database

Algorithms that automatically assess perceptual image quality are critical for numerous image processing applications. Recently, machine learning based blind image quality assessment has great progress, such as the BRISQUE [17], the LBIQ [18], the DIIVINE [19], and the BLIINDS [20] using SVR and the GRNN-based method [21]. Of these indices, the BRISQUE shows the best performance in overall, so here we use the same image features adopted by the BRISQUE index [17] to test our proposed MELM and M-WELM algorithms for image quality assessment (IQA).In reference [17], the authors used 36 natural scene statistical features in the spatial domain to predict image quality as shown in Table 2, that is, the shape and variance from a GGD fit of the MSCN coefficients, the shape, mean, left variance, and right variance from a GGD fit of the H pairwise products, V pairwise products, D1 pairwise products, and D2 pairwise products, which are extracted at two scales, the original image scale, and at a reduced resolution (low pass filtered and downsampled by a factor of 2). Here we also adopt these 36 image statistical features to predict image quality by using different ELM algorithms and then compare our proposed modified ELM algorithms with the reported ELM algorithms.Table 2
Summary of natural scene statistical features extracted in the spatial domain [14].
Feature ID
Feature description
Computation procedure
f
1-f
2
Shape and variance
Fit GGD to MSCN coefficients
f
3–f
6
Shape, mean, left variance, and right variance
Fit AGGD to H pairwise products
f
7–f
10
Shape, mean, left variance, and right variance
Fit AGGD to V pairwise products
f
11–f
14
Shape, mean, left variance, and right variance
Fit AGGD to D1 pairwise products
f
15–f
18
Shape, mean, left variance, and right variance
Fit AGGD to D2 pairwise productsFirstly, we test out proposed algorithm on the LIVE IQA database [22], which consists of 29 reference images with 779 distorted images spanning five different distortion categories” JPEG2000 (JP2K) and JPEG compression, additive white Gaussian noise (WN), Gaussian blur (Blur), and a Rayleigh fast-fading channel simulation (FF). Each of the distorted images has an associated difference mean opinion score (DMOS) which represents the subjective quality of the image.Three performance metrics are used to evaluate the algorithms. The first is the Spearman rank ordered correlation coefficient (SROCC), which measures the prediction monotonicity of the quality index. The second is the root mean square error (RMSE). The third is the running time.Because learning based method requires a training stage in order to construct the relationship between the extracted statistical features and DMOS, we split the LIVE dataset into two nonoverlapping sets—a training set and a testing set. The training set consists of 80%, 50%, or 30% of the 29 reference images and their associated distorted versions, respectively, while the testing set consists of the remaining 20%, 50%, or 70% of the 29 reference images and their associated distorted versions. The regression models are trained on the training set and the results are then tested on the testing set. In order to ensure that the proposed method is robust across content and is not governed by the specific train-test split utilized, we repeat this random 80% train—20% test, 50% train—50% test, and 30% train—70% test split 1000 times on the LIVE dataset and evaluate the performance on each of these test sets. The median Spearman rank ordered correlation coefficient (SROCC), RMS, and running time across these 1000 train-test trials are reported in Tables3, 4, and 5. For comparison, the results of SVR and other current reported ELM algorithms are also listed in Tables 3–5, using the same random train-test procedure 1000 times. The SVR is implemented by utilizing the libSVM package [23]. The kernel used for SVR is the radial basis function (RBF) kernel, whose parameters are estimated using cross validation on the training set. Other ELM algorithms are implemented by us. The used number of hidden neurons of M-WELM, M-ELM, W-ELM, OS-ELM, and EOS-ELM is set to 120, and the number of hidden neurons of ELM, B-ELM, and C-ELM is set to 75.Table 3
Median Spearman’s rank ordered correlation coefficient (SROCC), RMS, and running time across 1000 train-test combinations on the LIVE IQA database (80% samples are used for training).
Algorithm
SROCC (%)
RMS
Running time (s)
M-WELM
94.055
9.349
38.33
W-ELM [6]
93.325
9.566
37.60
M-ELM
93.312
9.451
30.17
ELM
93.115
10.240
46.96
OS-ELM [2]
93.159
9.973
227.44
EOS-ELM [3]
93.207
10.189
173.14
B-ELM [12]
91.288
10.541
343.41
C-ELM [13]
92.508
11.109
150.83
SVR
93.95
9.767
1763.97Table 4
Median Spearman’s rank ordered correlation coefficient (SROCC), RMS, and running time across 1000 train-test combinations on the LIVE IQA database (5
0
% samples are used for training).
Algorithm
SROCC (%)
RMS
Running time (s)
M-WELM
92.319
10.471
51.15
W-ELM [6]
91.931
10.563
45.45
MELM
92.140
10.494
42.54
ELM
90.170
11.391
56.69
OS-ELM [2]
90.920
11.260
120.25
EOS-ELM [3]
91.017
11.231
119.55
B-ELM [12]
90.324
11.384
112.01
C-ELM [13]
88.251
12.117
115.05
SVR
90.263
11.796
857.91Table 5
Median Spearman’s rank ordered correlation coefficient (SROCC), RMS, and running time across 1000 train-test combinations on the LIVE IQA database (3
0
% samples are used for training).
Algorithm
SROCC (%)
RMS
Running time (s)
M-WELM
90.340
11.621
63.75
W-ELM [6]
89.891
11.712
53.70
MELM
90.069
11.700
54.38
ELM
87.859
12.966
49.87
OS-ELM [2]
88.180
12.885
265.61
EOS-ELM [3]
88.304
12.853
271.91
B-ELM [12]
88.156
13.179
62.48
C-ELM [13]
86.316
14.061
72.40
SVR
87.655
13.110
415.63As it can be seen from Tables3–5, compared with ELM (or WELM), our proposed M-ELM (or M-WELM) shows better subjective judgment no matter how much the percentage of samples is used, which suggests the validity of introducing the structural risk minimization principle. Our M-WELM algorithm shows the best performance against the other reported ELM algorithm and SVR algorithm, especially when using less training samples, which further demonstrates the effectiveness of integrating the structural risk minimization principle and the weight method into the ELM model. In addition, we can find that our proposed M-WELM is far faster than the SVR, which provides an effective real-time solution to IQA.


### 4.3. Test on the TID 2008 Database

To prove the promotion of the proposed M-WELM, we further test on the same (available) distortions in an alternate database—the TID2008 [24]. It consists of 25 reference images and 17 distortion types with 1700 distorted images. Of these 25 reference images only 24 are natural images, so we test our algorithm only on these 24 images. Here we use all 779 distorted images in the LIVE IQA database as the training set and the images in the TID 2008 as the testing set. We still repeat the random train-test procedure 1000 times and report the median SROCC, RMS, and running time as shown in Table 6. The values of parameters of every algorithm are the same as the used in Section 4.2.Table 6
Median Spearman’s rank ordered correlation coefficient (SROCC), RMS, and running time across 1000 train-test combinations on the LIVE IQA and TID 2008 database.
Algorithm
SROCC (%)
RMS
Running time (s)
M-WELM
89.012
0.713
35.01
W-ELM [6]
88.762
0.72
28.94
MELM
88.879
0.718
21.85
ELM
87.756
0.752
28.78
OS-ELM [2]
87.630
0.758
102.38
EOS-ELM [3]
87.720
0.758
102.83
B-ELM [12]
87.750
0.752
62.66
C-ELM [13]
86.761
0.758
112.58
SVR
89.637
0.702
213.72From Table6, we can find our proposed M-WELM showing the highest consistency with the subjective scores amongst all types of ELM algorithms, and it is also competitive with the SVR in the performance, but it is far faster than the SVR, which provides a real-time solution to IQA.


## 4.1. Test on the Benchmark Boston Housing Data Set

Boston Housing data, obtained from the UCI database, is a data set commonly used for measuring the performance of regression algorithm. It contains the information of 506 sets of commodity houses in Boston Housing, including 12 continuous characteristics, one discrete characteristic, and house prices [16]. The purpose of regression estimation is to predict the average house price by training part of the samples.In the experiments, the samples are randomly divided into two sample groups: random 70% of them for training and the remaining 30% for test. We repeat the random train-test procedure 100 times and calculate the mean square training and prediction error of every algorithm, and the experimental results of several algorithms are shown in Table1. We have adjusted the parameters for every algorithm so that every algorithm can get a pretty good result. The number of hidden neurons of M-WELM, M-ELM, W-ELM, OS-ELM, and EOS-ELM is set to 180, and the number of hidden neurons of ELM, B-ELM, and C-ELM is set to 65.Table 1
The experimental results of the several algorithms on the Boston Housing data set.
Algorithm
Training error
Prediction error
M-WELM
2.253
3.364
M-ELM
2.154
3.592
W-ELM [6]
2.391
3.624
ELM
2.723
4.090
OS-ELM [2]
2.538
3.912
EOS-ELM [3]
2.686
3.745
B-ELM [12]
2.930
3.999
C-ELM [13]
2.615
3.653It can be seen from Table1, for the Boston Housing data set from a real-world multi-input single-output system, that our proposed M-WELM algorithm shows the best prediction performance than other types of ELMs, and M-ELM ranks number 2, with both of which indicating the robustness of our proposed idea for modifying ELM algorithm.


## 4.2. Test on the LIVE IQA Database

Algorithms that automatically assess perceptual image quality are critical for numerous image processing applications. Recently, machine learning based blind image quality assessment has great progress, such as the BRISQUE [17], the LBIQ [18], the DIIVINE [19], and the BLIINDS [20] using SVR and the GRNN-based method [21]. Of these indices, the BRISQUE shows the best performance in overall, so here we use the same image features adopted by the BRISQUE index [17] to test our proposed MELM and M-WELM algorithms for image quality assessment (IQA).In reference [17], the authors used 36 natural scene statistical features in the spatial domain to predict image quality as shown in Table 2, that is, the shape and variance from a GGD fit of the MSCN coefficients, the shape, mean, left variance, and right variance from a GGD fit of the H pairwise products, V pairwise products, D1 pairwise products, and D2 pairwise products, which are extracted at two scales, the original image scale, and at a reduced resolution (low pass filtered and downsampled by a factor of 2). Here we also adopt these 36 image statistical features to predict image quality by using different ELM algorithms and then compare our proposed modified ELM algorithms with the reported ELM algorithms.Table 2
Summary of natural scene statistical features extracted in the spatial domain [14].
Feature ID
Feature description
Computation procedure
f
1-f
2
Shape and variance
Fit GGD to MSCN coefficients
f
3–f
6
Shape, mean, left variance, and right variance
Fit AGGD to H pairwise products
f
7–f
10
Shape, mean, left variance, and right variance
Fit AGGD to V pairwise products
f
11–f
14
Shape, mean, left variance, and right variance
Fit AGGD to D1 pairwise products
f
15–f
18
Shape, mean, left variance, and right variance
Fit AGGD to D2 pairwise productsFirstly, we test out proposed algorithm on the LIVE IQA database [22], which consists of 29 reference images with 779 distorted images spanning five different distortion categories” JPEG2000 (JP2K) and JPEG compression, additive white Gaussian noise (WN), Gaussian blur (Blur), and a Rayleigh fast-fading channel simulation (FF). Each of the distorted images has an associated difference mean opinion score (DMOS) which represents the subjective quality of the image.Three performance metrics are used to evaluate the algorithms. The first is the Spearman rank ordered correlation coefficient (SROCC), which measures the prediction monotonicity of the quality index. The second is the root mean square error (RMSE). The third is the running time.Because learning based method requires a training stage in order to construct the relationship between the extracted statistical features and DMOS, we split the LIVE dataset into two nonoverlapping sets—a training set and a testing set. The training set consists of 80%, 50%, or 30% of the 29 reference images and their associated distorted versions, respectively, while the testing set consists of the remaining 20%, 50%, or 70% of the 29 reference images and their associated distorted versions. The regression models are trained on the training set and the results are then tested on the testing set. In order to ensure that the proposed method is robust across content and is not governed by the specific train-test split utilized, we repeat this random 80% train—20% test, 50% train—50% test, and 30% train—70% test split 1000 times on the LIVE dataset and evaluate the performance on each of these test sets. The median Spearman rank ordered correlation coefficient (SROCC), RMS, and running time across these 1000 train-test trials are reported in Tables3, 4, and 5. For comparison, the results of SVR and other current reported ELM algorithms are also listed in Tables 3–5, using the same random train-test procedure 1000 times. The SVR is implemented by utilizing the libSVM package [23]. The kernel used for SVR is the radial basis function (RBF) kernel, whose parameters are estimated using cross validation on the training set. Other ELM algorithms are implemented by us. The used number of hidden neurons of M-WELM, M-ELM, W-ELM, OS-ELM, and EOS-ELM is set to 120, and the number of hidden neurons of ELM, B-ELM, and C-ELM is set to 75.Table 3
Median Spearman’s rank ordered correlation coefficient (SROCC), RMS, and running time across 1000 train-test combinations on the LIVE IQA database (80% samples are used for training).
Algorithm
SROCC (%)
RMS
Running time (s)
M-WELM
94.055
9.349
38.33
W-ELM [6]
93.325
9.566
37.60
M-ELM
93.312
9.451
30.17
ELM
93.115
10.240
46.96
OS-ELM [2]
93.159
9.973
227.44
EOS-ELM [3]
93.207
10.189
173.14
B-ELM [12]
91.288
10.541
343.41
C-ELM [13]
92.508
11.109
150.83
SVR
93.95
9.767
1763.97Table 4
Median Spearman’s rank ordered correlation coefficient (SROCC), RMS, and running time across 1000 train-test combinations on the LIVE IQA database (5
0
% samples are used for training).
Algorithm
SROCC (%)
RMS
Running time (s)
M-WELM
92.319
10.471
51.15
W-ELM [6]
91.931
10.563
45.45
MELM
92.140
10.494
42.54
ELM
90.170
11.391
56.69
OS-ELM [2]
90.920
11.260
120.25
EOS-ELM [3]
91.017
11.231
119.55
B-ELM [12]
90.324
11.384
112.01
C-ELM [13]
88.251
12.117
115.05
SVR
90.263
11.796
857.91Table 5
Median Spearman’s rank ordered correlation coefficient (SROCC), RMS, and running time across 1000 train-test combinations on the LIVE IQA database (3
0
% samples are used for training).
Algorithm
SROCC (%)
RMS
Running time (s)
M-WELM
90.340
11.621
63.75
W-ELM [6]
89.891
11.712
53.70
MELM
90.069
11.700
54.38
ELM
87.859
12.966
49.87
OS-ELM [2]
88.180
12.885
265.61
EOS-ELM [3]
88.304
12.853
271.91
B-ELM [12]
88.156
13.179
62.48
C-ELM [13]
86.316
14.061
72.40
SVR
87.655
13.110
415.63As it can be seen from Tables3–5, compared with ELM (or WELM), our proposed M-ELM (or M-WELM) shows better subjective judgment no matter how much the percentage of samples is used, which suggests the validity of introducing the structural risk minimization principle. Our M-WELM algorithm shows the best performance against the other reported ELM algorithm and SVR algorithm, especially when using less training samples, which further demonstrates the effectiveness of integrating the structural risk minimization principle and the weight method into the ELM model. In addition, we can find that our proposed M-WELM is far faster than the SVR, which provides an effective real-time solution to IQA.


## 4.3. Test on the TID 2008 Database

To prove the promotion of the proposed M-WELM, we further test on the same (available) distortions in an alternate database—the TID2008 [24]. It consists of 25 reference images and 17 distortion types with 1700 distorted images. Of these 25 reference images only 24 are natural images, so we test our algorithm only on these 24 images. Here we use all 779 distorted images in the LIVE IQA database as the training set and the images in the TID 2008 as the testing set. We still repeat the random train-test procedure 1000 times and report the median SROCC, RMS, and running time as shown in Table 6. The values of parameters of every algorithm are the same as the used in Section 4.2.Table 6
Median Spearman’s rank ordered correlation coefficient (SROCC), RMS, and running time across 1000 train-test combinations on the LIVE IQA and TID 2008 database.
Algorithm
SROCC (%)
RMS
Running time (s)
M-WELM
89.012
0.713
35.01
W-ELM [6]
88.762
0.72
28.94
MELM
88.879
0.718
21.85
ELM
87.756
0.752
28.78
OS-ELM [2]
87.630
0.758
102.38
EOS-ELM [3]
87.720
0.758
102.83
B-ELM [12]
87.750
0.752
62.66
C-ELM [13]
86.761
0.758
112.58
SVR
89.637
0.702
213.72From Table6, we can find our proposed M-WELM showing the highest consistency with the subjective scores amongst all types of ELM algorithms, and it is also competitive with the SVR in the performance, but it is far faster than the SVR, which provides a real-time solution to IQA.


## 5. Conclusion

Current reported ELM and weighted ELM algorithms are based on empirical risk minimization principle, which may easily lead to the overfitting risk during learning process. By introducing the structural risk minimization principle to the ELM and weighted ELM algorithms, we propose an improved (weighted) extreme learning machine algorithm (M-WELM and M-ELM) to solve the overfitting problem, which takes into account both the empirical risk and the structural risk simultaneously and adjusts the proportion of the two risks properly. Our experimental results show that the M-WELM outperforms the current reported ELM algorithms in IQA and also has competitive performance with the SVR, but it is far faster than the SVR, which provides an effective real-time solution to IQA.



---

*Source: 426152-2014-05-22.xml*