# An Improved Brain Storm Optimization with Differential Evolution Strategy for Applications of ANNs

Authors: Zijian Cao; Xinhong Hei; Lei Wang; Yuhui Shi; Xiaofeng Rong
Journal: Mathematical Problems in Engineering
 (2015)

---

## Abstract

Brain Storm Optimization (BSO) algorithm is a swarm intelligence algorithm inspired by human being’s behavior of brainstorming. The performance of BSO is maintained by the creating process of ideas, but when it cannot find a better solution for some successive iterations, the result will be so inefficient that the population might be trapped into local optima. In this paper, we propose an improved BSO algorithm with differential evolution strategy and new step size method. Firstly, differential evolution strategy is incorporated into the creating operator of ideas to allow BSO jump out of stagnation, owing to its strong searching ability. Secondly, we introduce a new step size control method that can better balance exploration and exploitation at different searching generations. Finally, the proposed algorithm is first tested on 14 benchmark functions of CEC 2005 and then is applied to train artificial neural networks. Comparative experimental results illustrate that the proposed algorithm performs significantly better than the original BSO.


---

## Body

## 1. Introduction

In the past few decades, swarm intelligence algorithms, which were derived from the concepts, principles, and mechanisms of nature-inspired computation, have attracted more and more attention from researchers. Though swarm intelligence is a new category of stochastic, population-based optimization algorithms compared with evolutionary algorithms (EAs) such as Genetic Algorithms (GA) [1], Evolutionary Programming (EP) [2], Evolutionary Strategies (ES) [3], Genetic Programming (GP) [4], and differential evolution (DE) [5], so far, a lot of swarm intelligence optimization algorithms, such as Particle Swarm Optimization (PSO) [6, 7], have been proposed to tackle many challenging complex computational problems in science and industry.Brain Storm Optimization (BSO) algorithm [8, 9] is a swarm intelligence algorithm inspired by human being’s behavior of brainstorming. Derived from human creative problem-solving process, BSO has achieved more and more attention and has been applied to the function of optimization, optimal satellite formation reconfiguration [10], the design of DC Brushless Motor [11], economic dispatch considering wind power [12], multiobjective optimization [13, 14], and so forth. Due to the flexibility and scalability of the original BSO algorithm [15], many new versions of BSOs have been designed and implemented. To improve the performance and reduce the computation burden of BSO algorithm, Zhan et al. [16] modified the grouping operator and the creating operator and designed a BSO variant named Modified BSO (MBSO). Cheng et al. [17] used two kinds of partial reinitialization strategies to improve the population diversity in BSO algorithm. Recently, Yang et al. [18] proposed advanced discussion mechanism-based brain storm optimization (ADMBSO), which incorporated intercluster and intracluster discussions into BSO to control global and local searching ability. However, optimization problems have become more and more complex, from simple unimodal functions to hybrid rotated shifted multimodal functions. To the best of our knowledge, all variants of BSO have not been tested on the complex benchmark functions of CEC2005 [19]. Hence, more effective improvement to the original BSO algorithms is always necessary.Among existing metaheuristics, differential evolution (DE) is a simple yet powerful global optimization technique with successful applications in various areas [5]. To achieve the most satisfactory optimization performance, an improved BSO algorithm with differential evolution strategy and new step size method named BSODE, which can maintain both the exploration ability and the diversity of population, is proposed in the paper. The improved algorithm will be first tested on 14 benchmark functions of CEC 2005 and then will be applied to train artificial neural networks.The rest of this paper is organized as follows. Section2 briefly introduces the original BSO algorithm and the basic DE operator. Section 3 describes the improved BSO with differential evolution strategy (BSODE). Section 4 presents the tests on 14 benchmark functions. The applications for artificial neural network are given in Section 5, followed by conclusions in Section 6.


## 2. Related Work

### 2.1. Brain Storm Optimization

The BSO algorithm is motivated by the philosophy of brainstorming, and brainstorming is a widely used tool for increasing creativity in organizations which has achieved wide acceptance as a mean of facilitating creative thinking [20]. A potential solution in the solution space represents an idea in BSO. BSO sticks to the rules of interchange of ideas by a team and uses clustering, replacing, and creating operators to produce global optimum generation by generation. In the procedure of BSO, firstly, N ideas are randomly initialized within the solution space, and then each idea is evaluated according to its fitness function. Next m points of cluster center are also randomly selected and initialized like N ideas, where m is less than N. As presented in [8, 9], the rest of the process in BSO can be described as follows.


#### 2.1.1. Clustering Individuals

Clustering is a process of grouping similar objects together, and, during each generation, all the ideas are clustered intom clusters according to idea (or individual) features. The best idea in each center is chosen as its cluster center, and the clustering operation can refine a search area. K-means is a popular algorithm used in clustering; herein it is used in clustering operation.


#### 2.1.2. Disrupting Cluster Center

Cluster center disrupting operation randomly chooses a cluster center and replaces it with a newly generated idea with a probability ofp_replace, which is also named as the replacing operation. The value ofp_replace is utilized to control the probability to replace a cluster center by a randomly generated solution. This is used to avoid premature convergence and help individuals “jump out” of the local optima.


#### 2.1.3. Creating Individuals

To maintain the diversity of population, a new idea (individual) can be generated based on one idea or two in one cluster or two, respectively. In the creating operation, BSO first randomly chooses one cluster or two according to a probability ofp_one. Then, in the basis of choosing one cluster or two, an idea of cluster center or a random idea is selected with a probability ofp_one_center andp_two_center. The selecting operation is defined below as (1)Xselect=Xione  clusterrand∗Xi1+1-rand∗Xi2two  clusters,where rand is a random value between 0 and 1. After choosing one idea or two, the selected idea(s) is updated according to(2)Xnew=Xselect+ξ∗normrnd0,1,where normrnd is the Gaussian random value with mean 0 and variance 1 and ξ is an adjusting factor slowing the convergence speed down as the evolution goes, which can be expressed as (3)ξ=rand∗log⁡sig⁡0.5∗max_iteration-current_iterationk,where rand is a random value between 0 and 1. The max_iteration and current_iteration denote the maximum number of iterations and current number of iterations, respectively. The logsig is a logarithmic sigmoid transfer function, and such form is beneficial to global search ability at the beginning of the evolution and enhances local search ability when the process is approaching to the end. k is a predefined parameter for changing slopes of the logsig function. The new created idea is evaluated, and if the fitness value is better than the current idea, the old idea will be replaced by the new one.


### 2.2. Differential Evolution

Differential evolution (DE) [5] is a population-based and stochastic optimization algorithm. Like other EAs, DE begins with an initial population vector containing a number of target individuals. The current generation evolves into the next generation through evolutionary operations repeatedly until the termination condition is attained. For the original DE, the mutation, crossover, and selection operators are defined as follows.


#### 2.2.1. Mutation Operation

The mutation operation of classical DE scheme (DE/rand/bin) can be summarized as follows:(4)Vik=Xr1k+F∗Xr2k-Xr3k,where k is the generation number, the indices r1, r2, and r3 are mutually exclusive integers randomly chosen from the range between 1 and N, and F is a mutation scaling factor which affects the differential variation between two individuals.


#### 2.2.2. Crossover Operation

After the process of mutation, crossover operation is used to each pair of the target vector in order to enhance the potential diversity of the population. DE applies a crossover operator onXi and Vi to generate the offspring individual Ui at the kth generation. The crossover operation is defined as(5)Uijk=Vijkif  rand≤CRXijkotherwise,where rand is a random value between 0 and 1 and CR is a parameter of crossover probability.


#### 2.2.3. Selection Operation

The greedy selection is employed by means of comparing a parent and its corresponding offspring. The selection operation at thekth generation is described as(6)Xik+1=UikiffUik≤fXikXikotherwise,where f(X) is the objective function value of the trial vector X.


## 2.1. Brain Storm Optimization

The BSO algorithm is motivated by the philosophy of brainstorming, and brainstorming is a widely used tool for increasing creativity in organizations which has achieved wide acceptance as a mean of facilitating creative thinking [20]. A potential solution in the solution space represents an idea in BSO. BSO sticks to the rules of interchange of ideas by a team and uses clustering, replacing, and creating operators to produce global optimum generation by generation. In the procedure of BSO, firstly, N ideas are randomly initialized within the solution space, and then each idea is evaluated according to its fitness function. Next m points of cluster center are also randomly selected and initialized like N ideas, where m is less than N. As presented in [8, 9], the rest of the process in BSO can be described as follows.


### 2.1.1. Clustering Individuals

Clustering is a process of grouping similar objects together, and, during each generation, all the ideas are clustered intom clusters according to idea (or individual) features. The best idea in each center is chosen as its cluster center, and the clustering operation can refine a search area. K-means is a popular algorithm used in clustering; herein it is used in clustering operation.


### 2.1.2. Disrupting Cluster Center

Cluster center disrupting operation randomly chooses a cluster center and replaces it with a newly generated idea with a probability ofp_replace, which is also named as the replacing operation. The value ofp_replace is utilized to control the probability to replace a cluster center by a randomly generated solution. This is used to avoid premature convergence and help individuals “jump out” of the local optima.


### 2.1.3. Creating Individuals

To maintain the diversity of population, a new idea (individual) can be generated based on one idea or two in one cluster or two, respectively. In the creating operation, BSO first randomly chooses one cluster or two according to a probability ofp_one. Then, in the basis of choosing one cluster or two, an idea of cluster center or a random idea is selected with a probability ofp_one_center andp_two_center. The selecting operation is defined below as (1)Xselect=Xione  clusterrand∗Xi1+1-rand∗Xi2two  clusters,where rand is a random value between 0 and 1. After choosing one idea or two, the selected idea(s) is updated according to(2)Xnew=Xselect+ξ∗normrnd0,1,where normrnd is the Gaussian random value with mean 0 and variance 1 and ξ is an adjusting factor slowing the convergence speed down as the evolution goes, which can be expressed as (3)ξ=rand∗log⁡sig⁡0.5∗max_iteration-current_iterationk,where rand is a random value between 0 and 1. The max_iteration and current_iteration denote the maximum number of iterations and current number of iterations, respectively. The logsig is a logarithmic sigmoid transfer function, and such form is beneficial to global search ability at the beginning of the evolution and enhances local search ability when the process is approaching to the end. k is a predefined parameter for changing slopes of the logsig function. The new created idea is evaluated, and if the fitness value is better than the current idea, the old idea will be replaced by the new one.


## 2.1.1. Clustering Individuals

Clustering is a process of grouping similar objects together, and, during each generation, all the ideas are clustered intom clusters according to idea (or individual) features. The best idea in each center is chosen as its cluster center, and the clustering operation can refine a search area. K-means is a popular algorithm used in clustering; herein it is used in clustering operation.


## 2.1.2. Disrupting Cluster Center

Cluster center disrupting operation randomly chooses a cluster center and replaces it with a newly generated idea with a probability ofp_replace, which is also named as the replacing operation. The value ofp_replace is utilized to control the probability to replace a cluster center by a randomly generated solution. This is used to avoid premature convergence and help individuals “jump out” of the local optima.


## 2.1.3. Creating Individuals

To maintain the diversity of population, a new idea (individual) can be generated based on one idea or two in one cluster or two, respectively. In the creating operation, BSO first randomly chooses one cluster or two according to a probability ofp_one. Then, in the basis of choosing one cluster or two, an idea of cluster center or a random idea is selected with a probability ofp_one_center andp_two_center. The selecting operation is defined below as (1)Xselect=Xione  clusterrand∗Xi1+1-rand∗Xi2two  clusters,where rand is a random value between 0 and 1. After choosing one idea or two, the selected idea(s) is updated according to(2)Xnew=Xselect+ξ∗normrnd0,1,where normrnd is the Gaussian random value with mean 0 and variance 1 and ξ is an adjusting factor slowing the convergence speed down as the evolution goes, which can be expressed as (3)ξ=rand∗log⁡sig⁡0.5∗max_iteration-current_iterationk,where rand is a random value between 0 and 1. The max_iteration and current_iteration denote the maximum number of iterations and current number of iterations, respectively. The logsig is a logarithmic sigmoid transfer function, and such form is beneficial to global search ability at the beginning of the evolution and enhances local search ability when the process is approaching to the end. k is a predefined parameter for changing slopes of the logsig function. The new created idea is evaluated, and if the fitness value is better than the current idea, the old idea will be replaced by the new one.


## 2.2. Differential Evolution

Differential evolution (DE) [5] is a population-based and stochastic optimization algorithm. Like other EAs, DE begins with an initial population vector containing a number of target individuals. The current generation evolves into the next generation through evolutionary operations repeatedly until the termination condition is attained. For the original DE, the mutation, crossover, and selection operators are defined as follows.


### 2.2.1. Mutation Operation

The mutation operation of classical DE scheme (DE/rand/bin) can be summarized as follows:(4)Vik=Xr1k+F∗Xr2k-Xr3k,where k is the generation number, the indices r1, r2, and r3 are mutually exclusive integers randomly chosen from the range between 1 and N, and F is a mutation scaling factor which affects the differential variation between two individuals.


### 2.2.2. Crossover Operation

After the process of mutation, crossover operation is used to each pair of the target vector in order to enhance the potential diversity of the population. DE applies a crossover operator onXi and Vi to generate the offspring individual Ui at the kth generation. The crossover operation is defined as(5)Uijk=Vijkif  rand≤CRXijkotherwise,where rand is a random value between 0 and 1 and CR is a parameter of crossover probability.


### 2.2.3. Selection Operation

The greedy selection is employed by means of comparing a parent and its corresponding offspring. The selection operation at thekth generation is described as(6)Xik+1=UikiffUik≤fXikXikotherwise,where f(X) is the objective function value of the trial vector X.


## 2.2.1. Mutation Operation

The mutation operation of classical DE scheme (DE/rand/bin) can be summarized as follows:(4)Vik=Xr1k+F∗Xr2k-Xr3k,where k is the generation number, the indices r1, r2, and r3 are mutually exclusive integers randomly chosen from the range between 1 and N, and F is a mutation scaling factor which affects the differential variation between two individuals.


## 2.2.2. Crossover Operation

After the process of mutation, crossover operation is used to each pair of the target vector in order to enhance the potential diversity of the population. DE applies a crossover operator onXi and Vi to generate the offspring individual Ui at the kth generation. The crossover operation is defined as(5)Uijk=Vijkif  rand≤CRXijkotherwise,where rand is a random value between 0 and 1 and CR is a parameter of crossover probability.


## 2.2.3. Selection Operation

The greedy selection is employed by means of comparing a parent and its corresponding offspring. The selection operation at thekth generation is described as(6)Xik+1=UikiffUik≤fXikXikotherwise,where f(X) is the objective function value of the trial vector X.


## 3. The Improved Algorithm: BSODE

In order to achieve better performance, an improved BSO algorithm needs to make use of both the global search information about the search space and the local search information of solutions found so far. The global search information can guide the search for exploring promising areas, while the local search information of solutions can be helpful for exploitation. In this paper, we effectively integrate the mutation and crossover scheme of DE into BSO in the idea creating operator of intracluster and intercluster. The key reason for integrating differential evolution strategy into BSO is that it can take advantage of DE that is mainly based on the distance and direction information and has the advantage of not being biased towards any prior defined guider.Although the differential strategy has been utilized in [16] to design a BSO variant named Modified BSO (MBSO) and [10] to develop another BSO variant termed Close Loop BSO (CLBSO), our BSODE algorithm has significant differences with MBSO and CLBSO. On one hand, in the MBSO and CLBSO, a new idea Yi is created by adding the difference of two distinct random ideas Xr1 and Xr2 from all the current ideas to a current idea Xi according to(7)Yi=Xi+rand∗Xr1-Xr2,where rand is a random value between 0 and 1 and indices r1 and r2 are mutually exclusive integers randomly chosen from the range between 1 and N. Even though formula (7) is similar to the mutation formula (4) of DE algorithm, they are obviously different in essence. In formula (4), F, the mutation scaling factor, is a real constant between 0 and 2, and the idea behind F is to control the amplification of the differential variation. However, in formula (7), the factor, rand, represents a random direction scaling, and the scope of distance is less than F in some cases. In mutation operator, our BSODE will be strictly in accordance with formula (4) of DE algorithm. On the other hand, the MBSO and CLBSO do not take the crossover operator between ideas into account, and our BSODE uses the crossover operator to enhance the performance of local exploitation according to formula (5).Furthermore, in our BSODE, we mimic Osborn’s rule that different unusual ideas generating from different perspectives and suspending assumptions by participants are welcome in the brainstorming of human being. According to the inspiration, we add the differential evolution strategy into the normal idea to generate more different ideas. According to the above analysis, the differential evolution operators of BSODE algorithm include intracluster differential evolution operator and intercluster differential evolution operator. The detailed differential evolution operators of the BSODE algorithm are described below.


### 3.1. Differential Evolution Operator

#### 3.1.1. Intracluster Differential Evolution Operator

In creating operator of BSO, if the selected idea is a normal one in one cluster according to the probability ofp_one and (1,p_one_center), we will let the idea to learn from the differential value of two random selected ideas and the cluster center. The differential evolution operator of a normal idea is defined as(8)Xnew=Xcenter+F∗Xr1-Xr2,where F is the mutation scaling factor which affects the differential variation between two ideas and the indices r1 and r2 are mutually exclusive integers randomly chosen in the selected cluster. Then, according to formula (5), the crossover operation is used to generate new solutions by shuffling competing vectors and also to increase the diversity of the population.Figure1 shows the working process of intracluster differential evolution operator of a normal idea. The search space of the normal idea includes the area between two random ideas Xr1 and Xr2 and the cluster center Xcenter, and it can be seen that the local exploitation extends within the cluster space.Figure 1
Flowchart showing the working of intracluster differential evolution operator.


#### 3.1.2. Intercluster Differential Evolution Operator

In creating operator of BSO, if the selected ideas are two normal ones in two clusters according to the probability of (1,p_one) and (1,p_two_center), and we will let the idea to learn from the differential value of two randomly selected normal ideas in two clusters and the best idea in all clusters, respectively. The differential evolution operator of two normal ideas is defined as(9)Xnew=GobalIdea+F∗Xr1-Xr2,where F is the mutation scaling factor, GobalIdea is the best idea in all clusters, and Xr1 and Xr2 denote the normal idea of cluster 1 and cluster 2. Then, according to formula (5), the crossover operation is used to generate new solutions.Figure2 shows the working process of two normal ideas in intercluster differential evolution operator. The search space of the two normal ideas includes the area between global best idea point and two normal ideas, and it can be seen that the global exploration extends between the two clusters.Figure 2
Flowchart showing the working of intercluster differential evolution operator.According to the above analysis, the pseudocode of differential evolution operator about intracluster and intercluster is summarized in Algorithm1.Algorithm 1:Pseudocode of the differential evolution operator( ).
(1)     Begin
(2)      if rand( ) < p_one
(3)       Select a cluster according to the probabilities based on number of individuals
(4)       if rand( ) < p_one_center
(5)        execute original operator
(6)       else
(7)        execute intra-cluster DE operator to create Xnew using formula (8) and (5)
(8)       endif
(9)      else
(10)    Randomly select two clusters
(11)    if rand( ) < p_two_center
(12)     execute original operator
(13)    else
(14)     execute inter-cluster DE operator to create Xnew using formula (9) and (5)
(15)    endif
(16)   endif
(17)  end


### 3.2. A New Step Size Method

To adjust the convergence speed as the evolution goes in idea generation, the original BSO algorithm defines an adjusting factorξ described by formula (3). Figure 3(a) shows the adjusting factor which controls the scale of step. We can observe that at first the adjusting factor keeps around 1, while when half the number of generations has been reached, it rapidly turns to be near 0. This method to control the size of step can also balance exploration and exploitation at different searching generations. However, it just takes effect only for very short interval. Hence, we introduce a simple new step size method which is shown in Figure 3(b). The dynamic mutation function is described as follows:(10)ξ=rand∗exp⁡1-max_iterationmax_iteration-current_iteration+1,where rand is a random value between 0 and 1. The max_iteration and current_iteration denote the maximum number of iterations and current number of iterations, respectively.Figure 3
The comparison chart of adjusting factor between BSO and BSODE.
(a)
The curve of adjusting factor in BSO
(b)
The new step size adjusting factor in BSODE


### 3.3. Flowchart and Pseudocode of the BSODE

As previously analyzed, the complete flowchart of the BSODE algorithm is shown in Figure4.Figure 4
Flow chart of BSODE algorithm.This efficient differential evolution operator in intercluster improves the global search capability and avoids convergence to local minima. At the same time, the differential evolution operator in intracluster extends the capability of local exploitation and accelerates the convergence. The pseudocode of the BSODE is summarized in Algorithm2.Algorithm 2:Pseudocode of the BSODE( ).
(1)     Begin
(2)     Randomly initialize N ideas and evaluate their fitness
(3)     Initialize m cluster centers (m<N)
(4)     while (stopping condition not met)
(5)      Cluster N ideas into m clusters   %clustering operation
(6)      Record the best idea in each cluster as the cluster center
(7)      for (i=1 to N)          %creating operation
(8)       Execute Algorithm1: differential evolution operator()
(9)       Create Xnew1 using Xnew according formula (9)
(10)    Accept Xnew1 if f(Xnew1) is better than f(Xi)
(11)   endfor
(12)  endwhile
(13)  end


## 3.1. Differential Evolution Operator

### 3.1.1. Intracluster Differential Evolution Operator

In creating operator of BSO, if the selected idea is a normal one in one cluster according to the probability ofp_one and (1,p_one_center), we will let the idea to learn from the differential value of two random selected ideas and the cluster center. The differential evolution operator of a normal idea is defined as(8)Xnew=Xcenter+F∗Xr1-Xr2,where F is the mutation scaling factor which affects the differential variation between two ideas and the indices r1 and r2 are mutually exclusive integers randomly chosen in the selected cluster. Then, according to formula (5), the crossover operation is used to generate new solutions by shuffling competing vectors and also to increase the diversity of the population.Figure1 shows the working process of intracluster differential evolution operator of a normal idea. The search space of the normal idea includes the area between two random ideas Xr1 and Xr2 and the cluster center Xcenter, and it can be seen that the local exploitation extends within the cluster space.Figure 1
Flowchart showing the working of intracluster differential evolution operator.


### 3.1.2. Intercluster Differential Evolution Operator

In creating operator of BSO, if the selected ideas are two normal ones in two clusters according to the probability of (1,p_one) and (1,p_two_center), and we will let the idea to learn from the differential value of two randomly selected normal ideas in two clusters and the best idea in all clusters, respectively. The differential evolution operator of two normal ideas is defined as(9)Xnew=GobalIdea+F∗Xr1-Xr2,where F is the mutation scaling factor, GobalIdea is the best idea in all clusters, and Xr1 and Xr2 denote the normal idea of cluster 1 and cluster 2. Then, according to formula (5), the crossover operation is used to generate new solutions.Figure2 shows the working process of two normal ideas in intercluster differential evolution operator. The search space of the two normal ideas includes the area between global best idea point and two normal ideas, and it can be seen that the global exploration extends between the two clusters.Figure 2
Flowchart showing the working of intercluster differential evolution operator.According to the above analysis, the pseudocode of differential evolution operator about intracluster and intercluster is summarized in Algorithm1.Algorithm 1:Pseudocode of the differential evolution operator( ).
(1)     Begin
(2)      if rand( ) < p_one
(3)       Select a cluster according to the probabilities based on number of individuals
(4)       if rand( ) < p_one_center
(5)        execute original operator
(6)       else
(7)        execute intra-cluster DE operator to create Xnew using formula (8) and (5)
(8)       endif
(9)      else
(10)    Randomly select two clusters
(11)    if rand( ) < p_two_center
(12)     execute original operator
(13)    else
(14)     execute inter-cluster DE operator to create Xnew using formula (9) and (5)
(15)    endif
(16)   endif
(17)  end


## 3.1.1. Intracluster Differential Evolution Operator

In creating operator of BSO, if the selected idea is a normal one in one cluster according to the probability ofp_one and (1,p_one_center), we will let the idea to learn from the differential value of two random selected ideas and the cluster center. The differential evolution operator of a normal idea is defined as(8)Xnew=Xcenter+F∗Xr1-Xr2,where F is the mutation scaling factor which affects the differential variation between two ideas and the indices r1 and r2 are mutually exclusive integers randomly chosen in the selected cluster. Then, according to formula (5), the crossover operation is used to generate new solutions by shuffling competing vectors and also to increase the diversity of the population.Figure1 shows the working process of intracluster differential evolution operator of a normal idea. The search space of the normal idea includes the area between two random ideas Xr1 and Xr2 and the cluster center Xcenter, and it can be seen that the local exploitation extends within the cluster space.Figure 1
Flowchart showing the working of intracluster differential evolution operator.


## 3.1.2. Intercluster Differential Evolution Operator

In creating operator of BSO, if the selected ideas are two normal ones in two clusters according to the probability of (1,p_one) and (1,p_two_center), and we will let the idea to learn from the differential value of two randomly selected normal ideas in two clusters and the best idea in all clusters, respectively. The differential evolution operator of two normal ideas is defined as(9)Xnew=GobalIdea+F∗Xr1-Xr2,where F is the mutation scaling factor, GobalIdea is the best idea in all clusters, and Xr1 and Xr2 denote the normal idea of cluster 1 and cluster 2. Then, according to formula (5), the crossover operation is used to generate new solutions.Figure2 shows the working process of two normal ideas in intercluster differential evolution operator. The search space of the two normal ideas includes the area between global best idea point and two normal ideas, and it can be seen that the global exploration extends between the two clusters.Figure 2
Flowchart showing the working of intercluster differential evolution operator.According to the above analysis, the pseudocode of differential evolution operator about intracluster and intercluster is summarized in Algorithm1.Algorithm 1:Pseudocode of the differential evolution operator( ).
(1)     Begin
(2)      if rand( ) < p_one
(3)       Select a cluster according to the probabilities based on number of individuals
(4)       if rand( ) < p_one_center
(5)        execute original operator
(6)       else
(7)        execute intra-cluster DE operator to create Xnew using formula (8) and (5)
(8)       endif
(9)      else
(10)    Randomly select two clusters
(11)    if rand( ) < p_two_center
(12)     execute original operator
(13)    else
(14)     execute inter-cluster DE operator to create Xnew using formula (9) and (5)
(15)    endif
(16)   endif
(17)  end


## 3.2. A New Step Size Method

To adjust the convergence speed as the evolution goes in idea generation, the original BSO algorithm defines an adjusting factorξ described by formula (3). Figure 3(a) shows the adjusting factor which controls the scale of step. We can observe that at first the adjusting factor keeps around 1, while when half the number of generations has been reached, it rapidly turns to be near 0. This method to control the size of step can also balance exploration and exploitation at different searching generations. However, it just takes effect only for very short interval. Hence, we introduce a simple new step size method which is shown in Figure 3(b). The dynamic mutation function is described as follows:(10)ξ=rand∗exp⁡1-max_iterationmax_iteration-current_iteration+1,where rand is a random value between 0 and 1. The max_iteration and current_iteration denote the maximum number of iterations and current number of iterations, respectively.Figure 3
The comparison chart of adjusting factor between BSO and BSODE.
(a)
The curve of adjusting factor in BSO
(b)
The new step size adjusting factor in BSODE


## 3.3. Flowchart and Pseudocode of the BSODE

As previously analyzed, the complete flowchart of the BSODE algorithm is shown in Figure4.Figure 4
Flow chart of BSODE algorithm.This efficient differential evolution operator in intercluster improves the global search capability and avoids convergence to local minima. At the same time, the differential evolution operator in intracluster extends the capability of local exploitation and accelerates the convergence. The pseudocode of the BSODE is summarized in Algorithm2.Algorithm 2:Pseudocode of the BSODE( ).
(1)     Begin
(2)     Randomly initialize N ideas and evaluate their fitness
(3)     Initialize m cluster centers (m<N)
(4)     while (stopping condition not met)
(5)      Cluster N ideas into m clusters   %clustering operation
(6)      Record the best idea in each cluster as the cluster center
(7)      for (i=1 to N)          %creating operation
(8)       Execute Algorithm1: differential evolution operator()
(9)       Create Xnew1 using Xnew according formula (9)
(10)    Accept Xnew1 if f(Xnew1) is better than f(Xi)
(11)   endfor
(12)  endwhile
(13)  end


## 4. Benchmark Functions Optimization Problem

The numerical benchmark functions are used for comparing the performance and accuracy of our proposed BSODE algorithm with the original BSO algorithm and other variants of DE in this section. For a fair comparison, all the experiments are conducted on the same machine with an Intel 3.4 GHz CPU, 4 GB memory. The operating system is Windows 7 with MATLAB 8.0 (R2012b). For the purpose of reducing statistical errors, each function is independently run 25 times.


### 4.1. Benchmark Functions

We choose 14 widely known rotated or shifted benchmark functions used in CEC2005, which is given in Table1 [19]. All functions are tested on 30 dimensions. Results obtained are compared with some other variants of DE algorithms defined in the next subsection. The searching range and true optima for all functions are also given in Table 1. Among 14 benchmark functions, F1 to F5 are shifted unimodal functions, F6 to F12 are shifted or rotated multimodal functions, and F13 to F14 are expanded shifted or rotated functions.Table 1
Numerical benchmark function definition (CEC2005F1–F14).
Number
Function name
Type
Low range
Up range
Optima
F
1
Shifted sphere
Unimodal
−100
100
0
F
2
Shifted Schwefel
Unimodal
−100
100
0
F
3
Shifted rotated high conditioned elliptic
Unimodal
−100
100
0
F
4
Shifted Schwefel’s problem 1.2 with noise
Unimodal
−100
100
0
F
5
Schwefel’s problem 2.6
Unimodal
−100
100
0
F
6
Shifted Rosenbrock
Multimodal
−100
100
0
F
7
Shifted rotated Griewank
Multimodal
0
600
0
F
8
Shifted rotated Ackley
Multimodal
−32
32
0
F
9
Shifted Rastrigin
Multimodal
−5
5
0
F
10
Shifted rotated Rastrigin
Multimodal
−5
5
0
F
11
Shifted rotated Weierstrass
Multimodal
−0.5
0.5
0
F
12
Schwefel’s problem 2.13
Multimodal
−100
100
0
F
13
Expanded extended Griewank and Rosenbrock
Expanded
−3
1
0
F
14
Expanded rotated extended Scaffes
Expanded
−100
100
0


### 4.2. Parameter Settings of Comparative Algorithms

Because the BSODE algorithm combines the original BSO and the differential evolution strategy, we specially compare the performance of BSODE with the DE, the original BSO, and the PSO. Besides, we also compare the performance of BSODE with other variants of DE. We choose two DE variants, that is, CoDE [21] and SaDE [22]. To eliminate influences of statistical errors, each problem function is independently run 25 times which is a prescribed evaluation criterion in CEC2005 [19]. For all approaches, the population size is set to 30. The same stopping criterion is used in all algorithms, that is, reaching certain number of iterations or function evaluations (FEs). In our experiments, we have run all algorithms on the benchmark functions using the same FEs of 10000∗D for a fair comparison, where D is the size of the problem dimension. The parameter settings of all the algorithms are given in Table 2.Table 2
Algorithms parameter setting.
Number
Algorithm
Parameter setting
1
BSO-DE
M
=
5, p_one = 0.8, p_one_center = 0.4, and p_two_center = 0.5
2
BSO
M
=
5, p_replace = 0.2, p_one = 0.8, p_one_center = 0.4, p_two_center = 0.5, and K=20
3
PSO
ω: 0.9~0.4, c1=c2=1.49445, and global version
4
DE
F
=
0.5 and CR = 0.9
5
CoDE
F = [1.0 1.0 0.8] and CR = [0.1 0.9 0.2]
6
SaDE
F
i = Ni(0.5,0.3), CRi = Ni(0.5,0.1), and  p1=p2=0.5


### 4.3. The Contribution of the New Step Size Method to BSODE

To verify the rationality and effectivity of the new step size in BSODE and to fully understand the effect of the new step size method, herein, we investigate the contribution of the new step size to BSODE. Considering the consistency of the paper, we compare all 14 benchmark functions with 25 independent runs. We test the BSO, BSO with new step size (BSO-newstep), BSODE, and the BSODE with the original adjusting factor, that is, without new step size (BSODE-nonewstep). The experimental results are presented in Table3.Table 3
The results of solution accuracy for new step size.
Function
BSODE
BSO
BSO-newstep
BSODE-nonewstep
F
1
4.12
E
− 20  ±  5.77
E
− 20
+
2.65E  − 03  ±  3.03E  − 03
+
2.60E  − 16  ±  4.09E  − 16
+
1.31E  − 11  ±  6.35E  − 11
F
2
5.19
E
− 07  ±  3.87
E
− 07
+
2.76E + 00  ±  8.16E  − 01
+
4.08E  − 05  ±  1.34E  − 05
+
1.58E + 00  ±  5.86E  − 01
F
3
5.73
E
+ 05  ±  2.37
E
+ 05
+
1.33E + 06  ±  3.58E + 05
+
8.39E + 05  ±  3.03E + 05
+
1.67E + 06  ±  5.61E + 05
F
4
2.09E  − 01  ±  1.73E  – 01
+
1.29E + 04  ±  9.06E + 03
=
1.29
E
− 01  ±  1.15
E
− 01
+
1.98E + 03  ±  4.05E + 03
F
5
2.92
E
+ 04  ±  3.48
E
+ 02
=
3.12E + 04  ±  6.02E + 02
+
3.02E + 04  ±  5.01E + 02
=
2.93E + 04  ±  2.90E + 02
F
6
1.55E + 03  ±  2.64E + 03
=
1.52
E
+ 03  ±  2.71
E
+ 03
+
1.86E + 03  ±  2.66E + 03
+
1.77E + 03  ±  3.78E + 03
F
7
8.36
E
− 03  ±  1.05
E
− 02
+
5.35E + 03  ±  2.32E + 02
=
5.07E + 03  ±  1.18E + 02
=
5.45E  − 02  ±  3.19E  − 02
F
8
2.02
E
+ 01  ±  6.69
E
− 02
=
2.05E + 01  ±  1.17E  − 01
=
2.02
E
+ 01  ±  2.74
E
− 02
=
2.04E + 01  ±  1.10E  − 01
F
9
4.26E + 01  ±  1.37E + 01
+
4.97E + 01  ±  1.02E + 01
−
2.45
E
+ 01  ±  7.20
E
+ 00
+
8.67E + 01  ±  1.23E + 01
F
10
3.47
E
+ 01  ±  1.21
E
+ 01
+
4.59E + 01  ±  1.19E + 01
=
3.49E + 01  ±  5.54E + 00
+
8.50E + 01  ±  2.19E + 01
F
11
1.40
E
+ 01  ±  2.00
E
+ 00
+
2.43E + 01  ±  2.58E + 00
+
1.60E + 01  ±  1.98E + 00
+
2.69E + 01  ±  2.72E + 00
F
12
7.34E + 03  ±  7.32E + 03
+
3.16E + 04  ±  1.70E + 04
=
6.81E + 03  ±  3.50E + 03
+
3.61E + 04  ±  2.33E + 04
F
13
3.88
E
+ 00  ±  8.18
E
− 01
+
5.56E + 00  ±  1.35E + 00
+
4.70E + 00  ±  4.33E  − 01
=
5.30E + 00  ±  2.05E + 00
F
14
1.29E + 01  ±  4.23E− 01
=
1.32E + 01  ±  3.92E  − 01
=
1.26
E
+ 01  ±  4.44
E
− 01
=
1.33E + 01  ±  3.11E  − 01
+
/
=
/
-
10/4/0
7/6/1
9/5/0Table3 shows that the BSO-newstep (BSO with new step size) has better performance than the original BSO and BSODE has better performance than the BSODE-nonewstep (BSODE without new step size), and it can be concluded that the new step size plays a significant role in BSODE algorithm. This is in good agreement with our analysis in Section 3.2 that the control of the new step size can balance exploration and exploitation at different searching generations in Section 3.2.


### 4.4. Comparison of Results

#### 4.4.1. Comparisons on Solution Accuracy

The results of solution accuracy are given in Table4 in terms of the mean optimum solution and the standard deviation of the solutions obtained in the 25 independent runs by each algorithm over 300,000 FEs on 14 benchmark functions. In all experiments, the dimensions of all problems are 30. In each row of the table, the mean values are listed in the first part, and the standard deviations are listed in the last part, and the two parts are divided with a symbol “±.” The best results among the algorithms are displayed in bold.Table 4
The results of solution accuracy.
Function
BSODE
BSO
PSO
DE
CoDE
SaDE
F
1
4.12
E
-
20
±
5.77
E
-
20
2.65
E
-
03
±
3.03
E
-
03
4.67
E
+
03
±
2.86
E
+
03
5.46
E
-
01
±
2.59
E
+
00
0.00
E
+
00
±
0.00
E
+
00
0.00
E
+
00
±
0.00
E
+
00
F
2
5.19
E
-
07
±
3.87
E
-
07
2.76
E
+
00
±
8.16
E
-
01
6.51
E
+
03
±
8.15
E
+
03
1.19
E
-
01
±
5.88
E
-
01
9.11
E
-
06
±
1.11
E
-
05
5.55
E
-
10
±
6.99
E
-
10
F
3
5.73
E
+
05
±
2.37
E
+
05
1.33
E
+
06
±
3.58
E
+
05
2.29
E
+
07
±
4.93
E
+
07
3.40
E
+
05
±
1.69
E
+
05
3.65
E
+
05
±
1.60
E
+
05
3.67
E
+
05
±
1.82
E
+
05
F
4
2.09
E
-
01
±
1.73
E
-
01
1.29
E
+
04
±
9.06
E
+
03
4.56
E
+
03
±
6.54
E
+
03
1.25
E
+
02
±
5.87
E
+
02
2.01
E
+
01
±
5.61
E
+
01
4.42
E
+
01
±
1.25
E
+
02
F
5
2.92
E
+
04
±
3.48
E
+
02
3.12
E
+
04
±
6.02
E
+
02
3.43
E
+
04
±
2.28
E
+
03
2.90
E
+
04
±
2.17
E
-
01
2.9
3
E
+
04
±
2.21
E
-
04
2.93
E
+
04
±
1.09
E
+
02
F
6
1.55
E
+
03
±
2.64
E
+
03
1.52
E
+
03
±
2.71
E
+
03
8.15
E
+
08
±
9.30
E
+
08
5.43
E
+
06
±
1.51
E
+
07
3.45
E
+
00
±
3.67
E
+
00
3.07
E
+
01
±
3.24
E
+
01
F
7
8.36
E
-
03
±
1.05
E
-
02
5.35
E
+
03
±
2.32
E
+
02
5.36
E
+
03
±
5.34
E
+
02
4.70
E
+
03
±
9.47
E
-
13
4.70
E
+
03
±
1.30
E
-
12
4.70
E
+
03
±
4.15
E
-
13
F
8
2.02
E
+
01
±
6.69
E
-
02
2.05
E
+
01
±
1.17
E
-
01
2.09
E
+
01
±
6.56
E
-
02
2.09
E
+
01
±
7.37
E
-
02
2.05
E
+
01
±
3.25
E
-
01
2.09
E
+
01
±
5.36
E
-
02
F
9
4.26
E
+
01
±
1.37
E
+
01
4.97
E
+
01
±
1.02
E
+
01
6.90
E
+
01
±
2.28
E
+
01
2.18
E
+
01
±
5.63
E
+
00
1.34
E
-
12
±
3.35
E
-
12
1.95
E
+
00
±
1.27
E
+
00
F
10
3.47
E
+
01
±
1.21
E
+
01
4.59
E
+
01
±
1.19
E
+
01
1.26
E
+
02
±
3.82
E
+
01
4.28
E
+
01
±
1.59
E
+
01
3.98
E
+
01
±
1.55
E
+
01
4.75
E
+
01
±
1.13
E
+
01
F
11
1.40
E
+
01
±
2.00
E
+
00
2.43
E
+
01
±
2.58
E
+
00
2.02
E
+
01
±
2.59
E
+
00
2.30
E
+
01
±
6.15
E
+
00
1.05
E
+
01
±
4.09
E
+
00
1.65
E
+
01
±
2.85
E
+
00
F
12
7.34
E
+
03
±
7.32
E
+
03
3.16
E
+
04
±
1.70
E
+
04
4.83
E
+
03
±
5.14
E
+
03
2.08
E
+
05
±
8.19
E
+
04
1.31
E
+
05
±
2.54
E
+
04
9.67
E
+
04
±
1.47
E
+
04
F
13
3.88
E
+
00
±
8.18
E
-
01
5.56
E
+
00
±
1.35
E
+
00
3.65
E
+
00
±
1.14
E
+
00
3.24
E
+
00
±
5.77
E
-
01
4.02
E
+
00
±
1.22
E
+
00
2.76
E
+
00
±
3.60
E
-
01
F
14
1.29
E
+
01
±
4.23
E
-
01
1.32
E
+
01
±
3.92
E
-
01
1.21
E
+
01
±
5.27
E
-
01
1.30
E
+
01
±
2.66
E
-
01
1.25
E
+
01
±
3.82
E
-
01
1.27
E
+
01
±
1.83
E
-
01From Table4, it can be observed that the mean value and the standard deviation value of the original BSO algorithm performs well only for function F6 compared with the BSODE. The BSODE algorithm performs better than the other algorithms for 4 functions F4, F7, F8, and F10, especially for F4, and the mean value and the standard deviation value are far better than the other five algorithms. PSO outperforms the other algorithms in terms of solution accuracy only for function F12, and DE outperforms the other algorithms in terms of solution accuracy only for functions F3 and F5. Although CoDE performs better for 5 functions F1, F6, F9, F11, and F14 and SaDE performs better for 3 functions F1, F2, and 13, we can conclude that the BSODE algorithm remains still a good performance of the solution accuracy for complex shifted and rotated benchmark functions.


#### 4.4.2. The Comparison Results of Convergence Speed

Figure5 shows the convergence graphs of the compared 6 algorithms in terms of the mean fitness values of 25 runs. Due to page limitation, only graphs for 3 functions F1, F7, and F14 which include shifted, shifted rotated, and expanded rotated extended functions are shown. From Figure 5, we can observe that BSODE has fast convergence speed compared to the original BSO or similar convergence speed compared to the other four algorithms.Figure 5
Convergence performance of the 6 algorithms on parts of functions.
(a)
F
1, shifted sphere
(b)
F
7, shifted rotated Griewank
(c)
F
14, expanded rotated extended Scaffes


#### 4.4.3. The Comparison Results oft-Tests

To statistically compare BSODE with the other five algorithms,t-test [23, 24] is carried out to show the differences between BSODE and the other five algorithms. The t values on every function by two-tailed test with a significance level of 0.05 between BSODE and other algorithms are given in Table 5. Better, same, and worse give the number of experiments that BSODE performs significantly better than, almost the same as, and significantly worse than the compared algorithm under the same conditions, respectively.Table 5
Comparisons between BSODE and other algorithms ont-tests.
BSODE versus
BSO
PSO
DE
CoDE
SaDE
F
1
t value
6.18
11.53
1.49
−6.49
−6.49
p value
0.00007
0
0.29712
0.00003
0.00003
h value
1
1
0
1
1
F
2
t value
23.91
5.65
1.43
5.69
−9.07
p value
0
0.00022
0.31701
0.002
0
h value
1
1
1
1
1
F
3
t value
10.64
3.20
−7.38
−6.92
6.62
p value
0
0.02842
0
0
0.00002
h value
1
1
1
1
0
F
4
t value
10.04
4.93
1.50
2.51
2.50
p value
0
0.00105
0.0291
0.03137
0.0233
h value
1
1
1
1
1
F
5
t value
21.75
15.77
−7.99
7.99
2.49
p value
0
0
0
0
0.08411
h value
1
1
1
0
0
F
6
t value
−0.32
6.20
2.54
−4.64
−4.56
p value
0.08210
0.00006
0.0790
0.00194
0.00227
h value
0
1
0
1
1
F
7
t value
162.80
70.98
3165211.38
3165211.38
3165211.38
p value
0
0
0
0
0
h value
1
1
1
1
1
F
8
t value
11.70
43.94
43.35
6.73
50.07
p value
0
0
0
0.00002
0
h value
1
1
1
1
1
F
9
t value
8.59
10.28
−8.30
−28.45
−26.50
p value
0
0
0
0
0
h value
1
1
1
1
1
F
10
t value
5.10
16.34
3.05
1.96
5.98
p value
0.00074
0
0.03618
0.01729
0.0001
h value
1
1
1
1
1
F
11
t value
20.95
12.61
9.62
−5.38
4.73
p value
0
0
0
0.0004
0.00162
h value
1
1
1
1
1
F
12
t value
4.61
−2.53
16.25
24.62
21.93
p value
0.00205
0.08008
0
0
0
h value
1
0
1
1
1
F
13
t value
10.15
1.45
−1.02
3.25
−5.51
p value
0
0.31127
0.47521
0.02607
0.00031
h value
1
0
0
1
1
F
14
t value
4.35
−7.05
2.73
3.53
1.01
p value
0.00344
0.00001
0.05952
0.01618
0.47943
h value
1
1
0
0
0
Better
13
11
7
7
6
Same
0
2
3
2
3
Worse
1
1
4
5
5Table5 gives information about the t value, p value, and h value of 25 independent runs of 6 algorithms over 300,000 FEs on 14 test functions. The best results among the algorithms are shown in bold.For example, comparing BSODE and DE, the former significantly outperformed the latter in 7 functions (F2, F4, F7, F8, F10, F11, and F12), does the same as the latter in 3 functions (F6, F13, and F14), and does worse in 4 function (F1, F3, F5, and F9), yielding a “General Merit” value 7-4=3, indicating that the BSODE generally outperforms the DE. Although it performed slightly weaker in some functions, the BSODE in general offered much improved performance than other compared algorithms, as confirmed in Table 5.


## 4.1. Benchmark Functions

We choose 14 widely known rotated or shifted benchmark functions used in CEC2005, which is given in Table1 [19]. All functions are tested on 30 dimensions. Results obtained are compared with some other variants of DE algorithms defined in the next subsection. The searching range and true optima for all functions are also given in Table 1. Among 14 benchmark functions, F1 to F5 are shifted unimodal functions, F6 to F12 are shifted or rotated multimodal functions, and F13 to F14 are expanded shifted or rotated functions.Table 1
Numerical benchmark function definition (CEC2005F1–F14).
Number
Function name
Type
Low range
Up range
Optima
F
1
Shifted sphere
Unimodal
−100
100
0
F
2
Shifted Schwefel
Unimodal
−100
100
0
F
3
Shifted rotated high conditioned elliptic
Unimodal
−100
100
0
F
4
Shifted Schwefel’s problem 1.2 with noise
Unimodal
−100
100
0
F
5
Schwefel’s problem 2.6
Unimodal
−100
100
0
F
6
Shifted Rosenbrock
Multimodal
−100
100
0
F
7
Shifted rotated Griewank
Multimodal
0
600
0
F
8
Shifted rotated Ackley
Multimodal
−32
32
0
F
9
Shifted Rastrigin
Multimodal
−5
5
0
F
10
Shifted rotated Rastrigin
Multimodal
−5
5
0
F
11
Shifted rotated Weierstrass
Multimodal
−0.5
0.5
0
F
12
Schwefel’s problem 2.13
Multimodal
−100
100
0
F
13
Expanded extended Griewank and Rosenbrock
Expanded
−3
1
0
F
14
Expanded rotated extended Scaffes
Expanded
−100
100
0


## 4.2. Parameter Settings of Comparative Algorithms

Because the BSODE algorithm combines the original BSO and the differential evolution strategy, we specially compare the performance of BSODE with the DE, the original BSO, and the PSO. Besides, we also compare the performance of BSODE with other variants of DE. We choose two DE variants, that is, CoDE [21] and SaDE [22]. To eliminate influences of statistical errors, each problem function is independently run 25 times which is a prescribed evaluation criterion in CEC2005 [19]. For all approaches, the population size is set to 30. The same stopping criterion is used in all algorithms, that is, reaching certain number of iterations or function evaluations (FEs). In our experiments, we have run all algorithms on the benchmark functions using the same FEs of 10000∗D for a fair comparison, where D is the size of the problem dimension. The parameter settings of all the algorithms are given in Table 2.Table 2
Algorithms parameter setting.
Number
Algorithm
Parameter setting
1
BSO-DE
M
=
5, p_one = 0.8, p_one_center = 0.4, and p_two_center = 0.5
2
BSO
M
=
5, p_replace = 0.2, p_one = 0.8, p_one_center = 0.4, p_two_center = 0.5, and K=20
3
PSO
ω: 0.9~0.4, c1=c2=1.49445, and global version
4
DE
F
=
0.5 and CR = 0.9
5
CoDE
F = [1.0 1.0 0.8] and CR = [0.1 0.9 0.2]
6
SaDE
F
i = Ni(0.5,0.3), CRi = Ni(0.5,0.1), and  p1=p2=0.5


## 4.3. The Contribution of the New Step Size Method to BSODE

To verify the rationality and effectivity of the new step size in BSODE and to fully understand the effect of the new step size method, herein, we investigate the contribution of the new step size to BSODE. Considering the consistency of the paper, we compare all 14 benchmark functions with 25 independent runs. We test the BSO, BSO with new step size (BSO-newstep), BSODE, and the BSODE with the original adjusting factor, that is, without new step size (BSODE-nonewstep). The experimental results are presented in Table3.Table 3
The results of solution accuracy for new step size.
Function
BSODE
BSO
BSO-newstep
BSODE-nonewstep
F
1
4.12
E
− 20  ±  5.77
E
− 20
+
2.65E  − 03  ±  3.03E  − 03
+
2.60E  − 16  ±  4.09E  − 16
+
1.31E  − 11  ±  6.35E  − 11
F
2
5.19
E
− 07  ±  3.87
E
− 07
+
2.76E + 00  ±  8.16E  − 01
+
4.08E  − 05  ±  1.34E  − 05
+
1.58E + 00  ±  5.86E  − 01
F
3
5.73
E
+ 05  ±  2.37
E
+ 05
+
1.33E + 06  ±  3.58E + 05
+
8.39E + 05  ±  3.03E + 05
+
1.67E + 06  ±  5.61E + 05
F
4
2.09E  − 01  ±  1.73E  – 01
+
1.29E + 04  ±  9.06E + 03
=
1.29
E
− 01  ±  1.15
E
− 01
+
1.98E + 03  ±  4.05E + 03
F
5
2.92
E
+ 04  ±  3.48
E
+ 02
=
3.12E + 04  ±  6.02E + 02
+
3.02E + 04  ±  5.01E + 02
=
2.93E + 04  ±  2.90E + 02
F
6
1.55E + 03  ±  2.64E + 03
=
1.52
E
+ 03  ±  2.71
E
+ 03
+
1.86E + 03  ±  2.66E + 03
+
1.77E + 03  ±  3.78E + 03
F
7
8.36
E
− 03  ±  1.05
E
− 02
+
5.35E + 03  ±  2.32E + 02
=
5.07E + 03  ±  1.18E + 02
=
5.45E  − 02  ±  3.19E  − 02
F
8
2.02
E
+ 01  ±  6.69
E
− 02
=
2.05E + 01  ±  1.17E  − 01
=
2.02
E
+ 01  ±  2.74
E
− 02
=
2.04E + 01  ±  1.10E  − 01
F
9
4.26E + 01  ±  1.37E + 01
+
4.97E + 01  ±  1.02E + 01
−
2.45
E
+ 01  ±  7.20
E
+ 00
+
8.67E + 01  ±  1.23E + 01
F
10
3.47
E
+ 01  ±  1.21
E
+ 01
+
4.59E + 01  ±  1.19E + 01
=
3.49E + 01  ±  5.54E + 00
+
8.50E + 01  ±  2.19E + 01
F
11
1.40
E
+ 01  ±  2.00
E
+ 00
+
2.43E + 01  ±  2.58E + 00
+
1.60E + 01  ±  1.98E + 00
+
2.69E + 01  ±  2.72E + 00
F
12
7.34E + 03  ±  7.32E + 03
+
3.16E + 04  ±  1.70E + 04
=
6.81E + 03  ±  3.50E + 03
+
3.61E + 04  ±  2.33E + 04
F
13
3.88
E
+ 00  ±  8.18
E
− 01
+
5.56E + 00  ±  1.35E + 00
+
4.70E + 00  ±  4.33E  − 01
=
5.30E + 00  ±  2.05E + 00
F
14
1.29E + 01  ±  4.23E− 01
=
1.32E + 01  ±  3.92E  − 01
=
1.26
E
+ 01  ±  4.44
E
− 01
=
1.33E + 01  ±  3.11E  − 01
+
/
=
/
-
10/4/0
7/6/1
9/5/0Table3 shows that the BSO-newstep (BSO with new step size) has better performance than the original BSO and BSODE has better performance than the BSODE-nonewstep (BSODE without new step size), and it can be concluded that the new step size plays a significant role in BSODE algorithm. This is in good agreement with our analysis in Section 3.2 that the control of the new step size can balance exploration and exploitation at different searching generations in Section 3.2.


## 4.4. Comparison of Results

### 4.4.1. Comparisons on Solution Accuracy

The results of solution accuracy are given in Table4 in terms of the mean optimum solution and the standard deviation of the solutions obtained in the 25 independent runs by each algorithm over 300,000 FEs on 14 benchmark functions. In all experiments, the dimensions of all problems are 30. In each row of the table, the mean values are listed in the first part, and the standard deviations are listed in the last part, and the two parts are divided with a symbol “±.” The best results among the algorithms are displayed in bold.Table 4
The results of solution accuracy.
Function
BSODE
BSO
PSO
DE
CoDE
SaDE
F
1
4.12
E
-
20
±
5.77
E
-
20
2.65
E
-
03
±
3.03
E
-
03
4.67
E
+
03
±
2.86
E
+
03
5.46
E
-
01
±
2.59
E
+
00
0.00
E
+
00
±
0.00
E
+
00
0.00
E
+
00
±
0.00
E
+
00
F
2
5.19
E
-
07
±
3.87
E
-
07
2.76
E
+
00
±
8.16
E
-
01
6.51
E
+
03
±
8.15
E
+
03
1.19
E
-
01
±
5.88
E
-
01
9.11
E
-
06
±
1.11
E
-
05
5.55
E
-
10
±
6.99
E
-
10
F
3
5.73
E
+
05
±
2.37
E
+
05
1.33
E
+
06
±
3.58
E
+
05
2.29
E
+
07
±
4.93
E
+
07
3.40
E
+
05
±
1.69
E
+
05
3.65
E
+
05
±
1.60
E
+
05
3.67
E
+
05
±
1.82
E
+
05
F
4
2.09
E
-
01
±
1.73
E
-
01
1.29
E
+
04
±
9.06
E
+
03
4.56
E
+
03
±
6.54
E
+
03
1.25
E
+
02
±
5.87
E
+
02
2.01
E
+
01
±
5.61
E
+
01
4.42
E
+
01
±
1.25
E
+
02
F
5
2.92
E
+
04
±
3.48
E
+
02
3.12
E
+
04
±
6.02
E
+
02
3.43
E
+
04
±
2.28
E
+
03
2.90
E
+
04
±
2.17
E
-
01
2.9
3
E
+
04
±
2.21
E
-
04
2.93
E
+
04
±
1.09
E
+
02
F
6
1.55
E
+
03
±
2.64
E
+
03
1.52
E
+
03
±
2.71
E
+
03
8.15
E
+
08
±
9.30
E
+
08
5.43
E
+
06
±
1.51
E
+
07
3.45
E
+
00
±
3.67
E
+
00
3.07
E
+
01
±
3.24
E
+
01
F
7
8.36
E
-
03
±
1.05
E
-
02
5.35
E
+
03
±
2.32
E
+
02
5.36
E
+
03
±
5.34
E
+
02
4.70
E
+
03
±
9.47
E
-
13
4.70
E
+
03
±
1.30
E
-
12
4.70
E
+
03
±
4.15
E
-
13
F
8
2.02
E
+
01
±
6.69
E
-
02
2.05
E
+
01
±
1.17
E
-
01
2.09
E
+
01
±
6.56
E
-
02
2.09
E
+
01
±
7.37
E
-
02
2.05
E
+
01
±
3.25
E
-
01
2.09
E
+
01
±
5.36
E
-
02
F
9
4.26
E
+
01
±
1.37
E
+
01
4.97
E
+
01
±
1.02
E
+
01
6.90
E
+
01
±
2.28
E
+
01
2.18
E
+
01
±
5.63
E
+
00
1.34
E
-
12
±
3.35
E
-
12
1.95
E
+
00
±
1.27
E
+
00
F
10
3.47
E
+
01
±
1.21
E
+
01
4.59
E
+
01
±
1.19
E
+
01
1.26
E
+
02
±
3.82
E
+
01
4.28
E
+
01
±
1.59
E
+
01
3.98
E
+
01
±
1.55
E
+
01
4.75
E
+
01
±
1.13
E
+
01
F
11
1.40
E
+
01
±
2.00
E
+
00
2.43
E
+
01
±
2.58
E
+
00
2.02
E
+
01
±
2.59
E
+
00
2.30
E
+
01
±
6.15
E
+
00
1.05
E
+
01
±
4.09
E
+
00
1.65
E
+
01
±
2.85
E
+
00
F
12
7.34
E
+
03
±
7.32
E
+
03
3.16
E
+
04
±
1.70
E
+
04
4.83
E
+
03
±
5.14
E
+
03
2.08
E
+
05
±
8.19
E
+
04
1.31
E
+
05
±
2.54
E
+
04
9.67
E
+
04
±
1.47
E
+
04
F
13
3.88
E
+
00
±
8.18
E
-
01
5.56
E
+
00
±
1.35
E
+
00
3.65
E
+
00
±
1.14
E
+
00
3.24
E
+
00
±
5.77
E
-
01
4.02
E
+
00
±
1.22
E
+
00
2.76
E
+
00
±
3.60
E
-
01
F
14
1.29
E
+
01
±
4.23
E
-
01
1.32
E
+
01
±
3.92
E
-
01
1.21
E
+
01
±
5.27
E
-
01
1.30
E
+
01
±
2.66
E
-
01
1.25
E
+
01
±
3.82
E
-
01
1.27
E
+
01
±
1.83
E
-
01From Table4, it can be observed that the mean value and the standard deviation value of the original BSO algorithm performs well only for function F6 compared with the BSODE. The BSODE algorithm performs better than the other algorithms for 4 functions F4, F7, F8, and F10, especially for F4, and the mean value and the standard deviation value are far better than the other five algorithms. PSO outperforms the other algorithms in terms of solution accuracy only for function F12, and DE outperforms the other algorithms in terms of solution accuracy only for functions F3 and F5. Although CoDE performs better for 5 functions F1, F6, F9, F11, and F14 and SaDE performs better for 3 functions F1, F2, and 13, we can conclude that the BSODE algorithm remains still a good performance of the solution accuracy for complex shifted and rotated benchmark functions.


### 4.4.2. The Comparison Results of Convergence Speed

Figure5 shows the convergence graphs of the compared 6 algorithms in terms of the mean fitness values of 25 runs. Due to page limitation, only graphs for 3 functions F1, F7, and F14 which include shifted, shifted rotated, and expanded rotated extended functions are shown. From Figure 5, we can observe that BSODE has fast convergence speed compared to the original BSO or similar convergence speed compared to the other four algorithms.Figure 5
Convergence performance of the 6 algorithms on parts of functions.
(a)
F
1, shifted sphere
(b)
F
7, shifted rotated Griewank
(c)
F
14, expanded rotated extended Scaffes


### 4.4.3. The Comparison Results oft-Tests

To statistically compare BSODE with the other five algorithms,t-test [23, 24] is carried out to show the differences between BSODE and the other five algorithms. The t values on every function by two-tailed test with a significance level of 0.05 between BSODE and other algorithms are given in Table 5. Better, same, and worse give the number of experiments that BSODE performs significantly better than, almost the same as, and significantly worse than the compared algorithm under the same conditions, respectively.Table 5
Comparisons between BSODE and other algorithms ont-tests.
BSODE versus
BSO
PSO
DE
CoDE
SaDE
F
1
t value
6.18
11.53
1.49
−6.49
−6.49
p value
0.00007
0
0.29712
0.00003
0.00003
h value
1
1
0
1
1
F
2
t value
23.91
5.65
1.43
5.69
−9.07
p value
0
0.00022
0.31701
0.002
0
h value
1
1
1
1
1
F
3
t value
10.64
3.20
−7.38
−6.92
6.62
p value
0
0.02842
0
0
0.00002
h value
1
1
1
1
0
F
4
t value
10.04
4.93
1.50
2.51
2.50
p value
0
0.00105
0.0291
0.03137
0.0233
h value
1
1
1
1
1
F
5
t value
21.75
15.77
−7.99
7.99
2.49
p value
0
0
0
0
0.08411
h value
1
1
1
0
0
F
6
t value
−0.32
6.20
2.54
−4.64
−4.56
p value
0.08210
0.00006
0.0790
0.00194
0.00227
h value
0
1
0
1
1
F
7
t value
162.80
70.98
3165211.38
3165211.38
3165211.38
p value
0
0
0
0
0
h value
1
1
1
1
1
F
8
t value
11.70
43.94
43.35
6.73
50.07
p value
0
0
0
0.00002
0
h value
1
1
1
1
1
F
9
t value
8.59
10.28
−8.30
−28.45
−26.50
p value
0
0
0
0
0
h value
1
1
1
1
1
F
10
t value
5.10
16.34
3.05
1.96
5.98
p value
0.00074
0
0.03618
0.01729
0.0001
h value
1
1
1
1
1
F
11
t value
20.95
12.61
9.62
−5.38
4.73
p value
0
0
0
0.0004
0.00162
h value
1
1
1
1
1
F
12
t value
4.61
−2.53
16.25
24.62
21.93
p value
0.00205
0.08008
0
0
0
h value
1
0
1
1
1
F
13
t value
10.15
1.45
−1.02
3.25
−5.51
p value
0
0.31127
0.47521
0.02607
0.00031
h value
1
0
0
1
1
F
14
t value
4.35
−7.05
2.73
3.53
1.01
p value
0.00344
0.00001
0.05952
0.01618
0.47943
h value
1
1
0
0
0
Better
13
11
7
7
6
Same
0
2
3
2
3
Worse
1
1
4
5
5Table5 gives information about the t value, p value, and h value of 25 independent runs of 6 algorithms over 300,000 FEs on 14 test functions. The best results among the algorithms are shown in bold.For example, comparing BSODE and DE, the former significantly outperformed the latter in 7 functions (F2, F4, F7, F8, F10, F11, and F12), does the same as the latter in 3 functions (F6, F13, and F14), and does worse in 4 function (F1, F3, F5, and F9), yielding a “General Merit” value 7-4=3, indicating that the BSODE generally outperforms the DE. Although it performed slightly weaker in some functions, the BSODE in general offered much improved performance than other compared algorithms, as confirmed in Table 5.


## 4.4.1. Comparisons on Solution Accuracy

The results of solution accuracy are given in Table4 in terms of the mean optimum solution and the standard deviation of the solutions obtained in the 25 independent runs by each algorithm over 300,000 FEs on 14 benchmark functions. In all experiments, the dimensions of all problems are 30. In each row of the table, the mean values are listed in the first part, and the standard deviations are listed in the last part, and the two parts are divided with a symbol “±.” The best results among the algorithms are displayed in bold.Table 4
The results of solution accuracy.
Function
BSODE
BSO
PSO
DE
CoDE
SaDE
F
1
4.12
E
-
20
±
5.77
E
-
20
2.65
E
-
03
±
3.03
E
-
03
4.67
E
+
03
±
2.86
E
+
03
5.46
E
-
01
±
2.59
E
+
00
0.00
E
+
00
±
0.00
E
+
00
0.00
E
+
00
±
0.00
E
+
00
F
2
5.19
E
-
07
±
3.87
E
-
07
2.76
E
+
00
±
8.16
E
-
01
6.51
E
+
03
±
8.15
E
+
03
1.19
E
-
01
±
5.88
E
-
01
9.11
E
-
06
±
1.11
E
-
05
5.55
E
-
10
±
6.99
E
-
10
F
3
5.73
E
+
05
±
2.37
E
+
05
1.33
E
+
06
±
3.58
E
+
05
2.29
E
+
07
±
4.93
E
+
07
3.40
E
+
05
±
1.69
E
+
05
3.65
E
+
05
±
1.60
E
+
05
3.67
E
+
05
±
1.82
E
+
05
F
4
2.09
E
-
01
±
1.73
E
-
01
1.29
E
+
04
±
9.06
E
+
03
4.56
E
+
03
±
6.54
E
+
03
1.25
E
+
02
±
5.87
E
+
02
2.01
E
+
01
±
5.61
E
+
01
4.42
E
+
01
±
1.25
E
+
02
F
5
2.92
E
+
04
±
3.48
E
+
02
3.12
E
+
04
±
6.02
E
+
02
3.43
E
+
04
±
2.28
E
+
03
2.90
E
+
04
±
2.17
E
-
01
2.9
3
E
+
04
±
2.21
E
-
04
2.93
E
+
04
±
1.09
E
+
02
F
6
1.55
E
+
03
±
2.64
E
+
03
1.52
E
+
03
±
2.71
E
+
03
8.15
E
+
08
±
9.30
E
+
08
5.43
E
+
06
±
1.51
E
+
07
3.45
E
+
00
±
3.67
E
+
00
3.07
E
+
01
±
3.24
E
+
01
F
7
8.36
E
-
03
±
1.05
E
-
02
5.35
E
+
03
±
2.32
E
+
02
5.36
E
+
03
±
5.34
E
+
02
4.70
E
+
03
±
9.47
E
-
13
4.70
E
+
03
±
1.30
E
-
12
4.70
E
+
03
±
4.15
E
-
13
F
8
2.02
E
+
01
±
6.69
E
-
02
2.05
E
+
01
±
1.17
E
-
01
2.09
E
+
01
±
6.56
E
-
02
2.09
E
+
01
±
7.37
E
-
02
2.05
E
+
01
±
3.25
E
-
01
2.09
E
+
01
±
5.36
E
-
02
F
9
4.26
E
+
01
±
1.37
E
+
01
4.97
E
+
01
±
1.02
E
+
01
6.90
E
+
01
±
2.28
E
+
01
2.18
E
+
01
±
5.63
E
+
00
1.34
E
-
12
±
3.35
E
-
12
1.95
E
+
00
±
1.27
E
+
00
F
10
3.47
E
+
01
±
1.21
E
+
01
4.59
E
+
01
±
1.19
E
+
01
1.26
E
+
02
±
3.82
E
+
01
4.28
E
+
01
±
1.59
E
+
01
3.98
E
+
01
±
1.55
E
+
01
4.75
E
+
01
±
1.13
E
+
01
F
11
1.40
E
+
01
±
2.00
E
+
00
2.43
E
+
01
±
2.58
E
+
00
2.02
E
+
01
±
2.59
E
+
00
2.30
E
+
01
±
6.15
E
+
00
1.05
E
+
01
±
4.09
E
+
00
1.65
E
+
01
±
2.85
E
+
00
F
12
7.34
E
+
03
±
7.32
E
+
03
3.16
E
+
04
±
1.70
E
+
04
4.83
E
+
03
±
5.14
E
+
03
2.08
E
+
05
±
8.19
E
+
04
1.31
E
+
05
±
2.54
E
+
04
9.67
E
+
04
±
1.47
E
+
04
F
13
3.88
E
+
00
±
8.18
E
-
01
5.56
E
+
00
±
1.35
E
+
00
3.65
E
+
00
±
1.14
E
+
00
3.24
E
+
00
±
5.77
E
-
01
4.02
E
+
00
±
1.22
E
+
00
2.76
E
+
00
±
3.60
E
-
01
F
14
1.29
E
+
01
±
4.23
E
-
01
1.32
E
+
01
±
3.92
E
-
01
1.21
E
+
01
±
5.27
E
-
01
1.30
E
+
01
±
2.66
E
-
01
1.25
E
+
01
±
3.82
E
-
01
1.27
E
+
01
±
1.83
E
-
01From Table4, it can be observed that the mean value and the standard deviation value of the original BSO algorithm performs well only for function F6 compared with the BSODE. The BSODE algorithm performs better than the other algorithms for 4 functions F4, F7, F8, and F10, especially for F4, and the mean value and the standard deviation value are far better than the other five algorithms. PSO outperforms the other algorithms in terms of solution accuracy only for function F12, and DE outperforms the other algorithms in terms of solution accuracy only for functions F3 and F5. Although CoDE performs better for 5 functions F1, F6, F9, F11, and F14 and SaDE performs better for 3 functions F1, F2, and 13, we can conclude that the BSODE algorithm remains still a good performance of the solution accuracy for complex shifted and rotated benchmark functions.


## 4.4.2. The Comparison Results of Convergence Speed

Figure5 shows the convergence graphs of the compared 6 algorithms in terms of the mean fitness values of 25 runs. Due to page limitation, only graphs for 3 functions F1, F7, and F14 which include shifted, shifted rotated, and expanded rotated extended functions are shown. From Figure 5, we can observe that BSODE has fast convergence speed compared to the original BSO or similar convergence speed compared to the other four algorithms.Figure 5
Convergence performance of the 6 algorithms on parts of functions.
(a)
F
1, shifted sphere
(b)
F
7, shifted rotated Griewank
(c)
F
14, expanded rotated extended Scaffes


## 4.4.3. The Comparison Results oft-Tests

To statistically compare BSODE with the other five algorithms,t-test [23, 24] is carried out to show the differences between BSODE and the other five algorithms. The t values on every function by two-tailed test with a significance level of 0.05 between BSODE and other algorithms are given in Table 5. Better, same, and worse give the number of experiments that BSODE performs significantly better than, almost the same as, and significantly worse than the compared algorithm under the same conditions, respectively.Table 5
Comparisons between BSODE and other algorithms ont-tests.
BSODE versus
BSO
PSO
DE
CoDE
SaDE
F
1
t value
6.18
11.53
1.49
−6.49
−6.49
p value
0.00007
0
0.29712
0.00003
0.00003
h value
1
1
0
1
1
F
2
t value
23.91
5.65
1.43
5.69
−9.07
p value
0
0.00022
0.31701
0.002
0
h value
1
1
1
1
1
F
3
t value
10.64
3.20
−7.38
−6.92
6.62
p value
0
0.02842
0
0
0.00002
h value
1
1
1
1
0
F
4
t value
10.04
4.93
1.50
2.51
2.50
p value
0
0.00105
0.0291
0.03137
0.0233
h value
1
1
1
1
1
F
5
t value
21.75
15.77
−7.99
7.99
2.49
p value
0
0
0
0
0.08411
h value
1
1
1
0
0
F
6
t value
−0.32
6.20
2.54
−4.64
−4.56
p value
0.08210
0.00006
0.0790
0.00194
0.00227
h value
0
1
0
1
1
F
7
t value
162.80
70.98
3165211.38
3165211.38
3165211.38
p value
0
0
0
0
0
h value
1
1
1
1
1
F
8
t value
11.70
43.94
43.35
6.73
50.07
p value
0
0
0
0.00002
0
h value
1
1
1
1
1
F
9
t value
8.59
10.28
−8.30
−28.45
−26.50
p value
0
0
0
0
0
h value
1
1
1
1
1
F
10
t value
5.10
16.34
3.05
1.96
5.98
p value
0.00074
0
0.03618
0.01729
0.0001
h value
1
1
1
1
1
F
11
t value
20.95
12.61
9.62
−5.38
4.73
p value
0
0
0
0.0004
0.00162
h value
1
1
1
1
1
F
12
t value
4.61
−2.53
16.25
24.62
21.93
p value
0.00205
0.08008
0
0
0
h value
1
0
1
1
1
F
13
t value
10.15
1.45
−1.02
3.25
−5.51
p value
0
0.31127
0.47521
0.02607
0.00031
h value
1
0
0
1
1
F
14
t value
4.35
−7.05
2.73
3.53
1.01
p value
0.00344
0.00001
0.05952
0.01618
0.47943
h value
1
1
0
0
0
Better
13
11
7
7
6
Same
0
2
3
2
3
Worse
1
1
4
5
5Table5 gives information about the t value, p value, and h value of 25 independent runs of 6 algorithms over 300,000 FEs on 14 test functions. The best results among the algorithms are shown in bold.For example, comparing BSODE and DE, the former significantly outperformed the latter in 7 functions (F2, F4, F7, F8, F10, F11, and F12), does the same as the latter in 3 functions (F6, F13, and F14), and does worse in 4 function (F1, F3, F5, and F9), yielding a “General Merit” value 7-4=3, indicating that the BSODE generally outperforms the DE. Although it performed slightly weaker in some functions, the BSODE in general offered much improved performance than other compared algorithms, as confirmed in Table 5.


## 5. Applications of ANN Using BSODE

Artificial neural network (ANN) [25] simulates the learning pattern of human brain for solving many difficult problems in the field of applications such as prediction of principal ground-motion parameters [26] and prediction of peak ground acceleration [27]. ANN is a flexible mathematical scheme which is capable of identifying complex nonlinear relationships between input data and output data. The task to train ANN needs vigorous optimization techniques because the search space is highly dimensional and multimodal which suffers from noises and data in general. The usually used training method is Back-Propagation (BP) algorithm [28]. However, BP algorithm has some potential problems. On one hand, for the complex pattern classification problems [29], BP algorithm may lightly get trapped into local minima. On the other hand, the training time of the BP algorithm is very long for complex nonlinear problems.In recent decades, in order to deal with the complicated training problem of the ANN, many metaheuristic optimization algorithms, such as Simulated Annealing (SA) [26, 27], GA [30], and PSO [31], have been utilized to optimize the weights and biases of ANN. Besides, the recently proposed EvoNN [32, 33] algorithm, which utilizes the multiobjective optimization technique in the training process of a feed-forward neural network, ensures correct neural training by working out a Pareto tradeoff between the accuracy of the training and the complexity of the network. In this section, we use our BSODE algorithm to adjust connection weights and biases of ANN. To validate the usefulness and the efficiency of our BSODE algorithm, two types of applications including time series prediction and System Identification are illustrated.


### 5.1. Training ANN with BSODE Algorithm

The structure of training ANN using our BSODE algorithm is a three-layer feed-forward network. The basic structure of the proposed scheme is given in Figure6.Figure 6
BSODE-based ANN training model.In our experiments, activation function used in the input layer is a linear combination of input variables, activation function used in the hidden layer is the sigma function, and activation function used in the output layer is a linear combination of output variables of hidden layer. In training ANN, the aim is to find a set of weights and biases with the smallest error value. Herein, the objective function is the mean sum of squared errors (MSE) which is given as follows:(11)MSE=1Q∗K∑i=1Q∑jk12dij-yij2,where Q is the number of training data sets, K is the number of output units, dij is desired output, and yij is output inferred from ANN.


### 5.2. Time Series Prediction of Lorenz Chaotic System

The first application is the time series prediction of Lorenz chaotic system described by the following equation [34]:(12a)x˙t=ayt-xt(12b)y˙t=bxt-xtzt-yt(12c)z˙t=xtyt-czt,where a, b, and c are parameters of the Lorenz chaotic system. The system is chaotic when a=10, b=28, and c=8/3. The differential equations were solved numerically using 4th order Runge-Kutta integration with a step size 0.01 and initial values x(0)=0, y(0)=-1, and z(0)=0. The x-coordinate of the Lorenz time series system is chosen for prediction. The goal is to construct the prediction model of chaotic time series according to the following formula:(13)yt+1=fyt-18,yt-12,…,yt-6,yt.In order to train the ANN, we choose 10000 data sets according to the real model, where the first 8000 data sets are discarded; the rest normalized 2000 data sets are considered as experiment data. The former 1500 points are selected as the training data points and the rest 500 points are selected as the testing data points to test the validity of the model.In this experiment, the parameter settings of the six algorithms are the same as those in Section4.2, and the maximal generation used as ended condition of algorithm is 100 for BSODE, BSO, PSO, DE, CoDE, and SaDE. The results are shown in Table 6 in terms of the mean optimum solution, the standard deviation of the solutions, and run time independent runs by each algorithm.Table 6
Comparisons of MSE on Lorenz chaotic time series over 25 runs.
Performance
BSODE
BSO
PSO
DE
CoDE
SaDE
Training error
3
.
3
3
E
-
03
±
2
.
0
2
E
-
03
3.81
E
-
03
±
1.70
E
-
03
(
+
)
4.01
E
-
03
±
2.30
E
-
03
(
+
)
1
.
03
E
-
02
±
5
.
95
E
-
03
(
+
)
4.84
E
-
03
±
1.97
E
-
03
(
+
)
9.97
E
-
02
±
1.11
E
-
01
(
+
)
Testing error
3
.
4
8
E
-
03
±
2
.
1
5
E
-
02
3.99
E
-
03
±
1.78
E
-
03
(
+
)
4.21
E
-
03
±
3.20
E
-
03
(
+
)
1
.
0
6
E
-
02
±
6
.
0
8
E
-
03
(
+
)
4.99
E
-
03
±
2.07
E
-
03
(
+
)
9.74
E
-
02
±
1.13
E
-
01
(
+
)
Run time(s)
7
.
3
9
E
+
01
±
8
.
3
0
E
-
01
7
.
4
1
E
+
01
±
6.60
E
-
01
(
+
)
7.11
E
+
01
±
3.07
E
+
00
(
-
)
7
.
3
5
E
+
01
±
4
.
8
5
E
-
01
(
-
)
2.13
E
+
02
±
3.05
E
+
00
(
+
)
7.14
E
+
01
±
4.99
E
-
01
(
-
)As is seen in Table6, SaDE has the worst MSE in both training and testing cases. BSODE performs better than BSO, PSO, DE, CoDE, and SaDE in the mean of the MSE. The model trained by six algorithms follows the chaotic behavior of the Lorenz chaotic time series which is demonstrated in Figure 7. It can be concluded that BSODE is the most effective learning algorithm for Lorenz chaotic time series even though BSODE needs more running time than PSO algorithm.Figure 7
Comparison of the performance curves for Lorenz chaotic time series prediction.
(a)
The prediction results using BSODE
(b)
The prediction results using BSO
(c)
The prediction results using PSO
(d)
The prediction results using DE
(e)
The prediction results using CoDE
(f)
The prediction results using SaDE


### 5.3. Nonlinear System Identification

The second application is to build a three-layer feed-forward ANN with three input units, five hidden units, and one output unit. The ANN is used to model the curve of a nonlinear function which is given by the following equation:(14)yt+1=ytyt-1yt+2.51+y2t+y2t-1+utut=sin⁡2πt25.The simulation target is to construct the approximation model of the nonlinear function according to the following formula:(15)yt+1=fut,yt-1,yt.To train the ANN, we choose 200 pairs of data from the real model. In addition, to further evaluate the performance of BSODE algorithm, we add 20 db additive white Gaussian noise into the 200 pairs of data.In this experiment, population size of each algorithm is 30, and the maximal generation used as termination condition of each algorithm is 1000 for BSODE, BSO, PSO, DE, CoDE, and SaDE. The parameter settings of the six algorithms are also the same as in Section4.2. The results are given in Table 7 in terms of the mean MSE and the standard deviation obtained in the 25 independent runs for six algorithms.Table 7
Comparisons between BSODE and other algorithms on MSE over 25 runs.
App.
BSODE
BSO
PSO
DE
CoDE
SaDE
Without noise
6.69
E
-
03
±
3
.
8
9
E
-
03
7
.
7
8
E
-
03
±
3
.
87
E
-
03
(
+
)
2.90
E
-
03
±
1.69
E
-
03
(
-
)
9
.
89
E
-
03
±
1
.
6
9
E
-
02
(
+
)
3.69
E
-
03
±
6.36
E
-
03
(
-
)
1.68
E
-
03
±
1.41
E
-
03
(
-
)
With noise
3
.
20
E
-
02
±
3
.
5
5
E
-
03
4
.
08
E
-
02
±
6
.
2
4
E
-
03
(
+
)
3.28
E
-
02
±
3.66
E
-
03
(
+
)
3
.
2
7
E
-
02
±
1
.
5
5
E
-
02
(
+
)
3.63
E
-
02
±
4.14
E
-
03
(
+
)
3
.
5
1
E
-
02
±
4.40
E
-
03
(
+
)From Table7, in case without noise, we observed that BSODE performs worse than SaDE, CoDE, and PSO, but BSODE performs better than BSO and DE in terms of the mean MSE. However, in case of 20 db additive white Gaussian noise, BSODE performs better than all other five algorithms. Figure 8 shows that the output of the identified system tracks the target output for three algorithms. We can conclude that BSODE has the best performance in case without noise and a better performance compared with BSO and DE in building the approximation model.Figure 8
Comparison of the performance curves for System Identification.
(a)
System Identification without noise using BSODE
(b)
System Identification with noise using BSODE
(c)
System Identification without noise using BSO
(d)
System Identification with noise using BSO
(e)
System Identification without noise using PSO
(f)
System Identification with noise using PSO
(g)
System Identification without noise using DE
(h)
System Identification with noise using DE
(i)
System Identification without noise using CoDE
(j)
System Identification with noise using CoDE
(k)
System Identification without noise using SaDE
(l)
System Identification with noise using SaDE


## 5.1. Training ANN with BSODE Algorithm

The structure of training ANN using our BSODE algorithm is a three-layer feed-forward network. The basic structure of the proposed scheme is given in Figure6.Figure 6
BSODE-based ANN training model.In our experiments, activation function used in the input layer is a linear combination of input variables, activation function used in the hidden layer is the sigma function, and activation function used in the output layer is a linear combination of output variables of hidden layer. In training ANN, the aim is to find a set of weights and biases with the smallest error value. Herein, the objective function is the mean sum of squared errors (MSE) which is given as follows:(11)MSE=1Q∗K∑i=1Q∑jk12dij-yij2,where Q is the number of training data sets, K is the number of output units, dij is desired output, and yij is output inferred from ANN.


## 5.2. Time Series Prediction of Lorenz Chaotic System

The first application is the time series prediction of Lorenz chaotic system described by the following equation [34]:(12a)x˙t=ayt-xt(12b)y˙t=bxt-xtzt-yt(12c)z˙t=xtyt-czt,where a, b, and c are parameters of the Lorenz chaotic system. The system is chaotic when a=10, b=28, and c=8/3. The differential equations were solved numerically using 4th order Runge-Kutta integration with a step size 0.01 and initial values x(0)=0, y(0)=-1, and z(0)=0. The x-coordinate of the Lorenz time series system is chosen for prediction. The goal is to construct the prediction model of chaotic time series according to the following formula:(13)yt+1=fyt-18,yt-12,…,yt-6,yt.In order to train the ANN, we choose 10000 data sets according to the real model, where the first 8000 data sets are discarded; the rest normalized 2000 data sets are considered as experiment data. The former 1500 points are selected as the training data points and the rest 500 points are selected as the testing data points to test the validity of the model.In this experiment, the parameter settings of the six algorithms are the same as those in Section4.2, and the maximal generation used as ended condition of algorithm is 100 for BSODE, BSO, PSO, DE, CoDE, and SaDE. The results are shown in Table 6 in terms of the mean optimum solution, the standard deviation of the solutions, and run time independent runs by each algorithm.Table 6
Comparisons of MSE on Lorenz chaotic time series over 25 runs.
Performance
BSODE
BSO
PSO
DE
CoDE
SaDE
Training error
3
.
3
3
E
-
03
±
2
.
0
2
E
-
03
3.81
E
-
03
±
1.70
E
-
03
(
+
)
4.01
E
-
03
±
2.30
E
-
03
(
+
)
1
.
03
E
-
02
±
5
.
95
E
-
03
(
+
)
4.84
E
-
03
±
1.97
E
-
03
(
+
)
9.97
E
-
02
±
1.11
E
-
01
(
+
)
Testing error
3
.
4
8
E
-
03
±
2
.
1
5
E
-
02
3.99
E
-
03
±
1.78
E
-
03
(
+
)
4.21
E
-
03
±
3.20
E
-
03
(
+
)
1
.
0
6
E
-
02
±
6
.
0
8
E
-
03
(
+
)
4.99
E
-
03
±
2.07
E
-
03
(
+
)
9.74
E
-
02
±
1.13
E
-
01
(
+
)
Run time(s)
7
.
3
9
E
+
01
±
8
.
3
0
E
-
01
7
.
4
1
E
+
01
±
6.60
E
-
01
(
+
)
7.11
E
+
01
±
3.07
E
+
00
(
-
)
7
.
3
5
E
+
01
±
4
.
8
5
E
-
01
(
-
)
2.13
E
+
02
±
3.05
E
+
00
(
+
)
7.14
E
+
01
±
4.99
E
-
01
(
-
)As is seen in Table6, SaDE has the worst MSE in both training and testing cases. BSODE performs better than BSO, PSO, DE, CoDE, and SaDE in the mean of the MSE. The model trained by six algorithms follows the chaotic behavior of the Lorenz chaotic time series which is demonstrated in Figure 7. It can be concluded that BSODE is the most effective learning algorithm for Lorenz chaotic time series even though BSODE needs more running time than PSO algorithm.Figure 7
Comparison of the performance curves for Lorenz chaotic time series prediction.
(a)
The prediction results using BSODE
(b)
The prediction results using BSO
(c)
The prediction results using PSO
(d)
The prediction results using DE
(e)
The prediction results using CoDE
(f)
The prediction results using SaDE


## 5.3. Nonlinear System Identification

The second application is to build a three-layer feed-forward ANN with three input units, five hidden units, and one output unit. The ANN is used to model the curve of a nonlinear function which is given by the following equation:(14)yt+1=ytyt-1yt+2.51+y2t+y2t-1+utut=sin⁡2πt25.The simulation target is to construct the approximation model of the nonlinear function according to the following formula:(15)yt+1=fut,yt-1,yt.To train the ANN, we choose 200 pairs of data from the real model. In addition, to further evaluate the performance of BSODE algorithm, we add 20 db additive white Gaussian noise into the 200 pairs of data.In this experiment, population size of each algorithm is 30, and the maximal generation used as termination condition of each algorithm is 1000 for BSODE, BSO, PSO, DE, CoDE, and SaDE. The parameter settings of the six algorithms are also the same as in Section4.2. The results are given in Table 7 in terms of the mean MSE and the standard deviation obtained in the 25 independent runs for six algorithms.Table 7
Comparisons between BSODE and other algorithms on MSE over 25 runs.
App.
BSODE
BSO
PSO
DE
CoDE
SaDE
Without noise
6.69
E
-
03
±
3
.
8
9
E
-
03
7
.
7
8
E
-
03
±
3
.
87
E
-
03
(
+
)
2.90
E
-
03
±
1.69
E
-
03
(
-
)
9
.
89
E
-
03
±
1
.
6
9
E
-
02
(
+
)
3.69
E
-
03
±
6.36
E
-
03
(
-
)
1.68
E
-
03
±
1.41
E
-
03
(
-
)
With noise
3
.
20
E
-
02
±
3
.
5
5
E
-
03
4
.
08
E
-
02
±
6
.
2
4
E
-
03
(
+
)
3.28
E
-
02
±
3.66
E
-
03
(
+
)
3
.
2
7
E
-
02
±
1
.
5
5
E
-
02
(
+
)
3.63
E
-
02
±
4.14
E
-
03
(
+
)
3
.
5
1
E
-
02
±
4.40
E
-
03
(
+
)From Table7, in case without noise, we observed that BSODE performs worse than SaDE, CoDE, and PSO, but BSODE performs better than BSO and DE in terms of the mean MSE. However, in case of 20 db additive white Gaussian noise, BSODE performs better than all other five algorithms. Figure 8 shows that the output of the identified system tracks the target output for three algorithms. We can conclude that BSODE has the best performance in case without noise and a better performance compared with BSO and DE in building the approximation model.Figure 8
Comparison of the performance curves for System Identification.
(a)
System Identification without noise using BSODE
(b)
System Identification with noise using BSODE
(c)
System Identification without noise using BSO
(d)
System Identification with noise using BSO
(e)
System Identification without noise using PSO
(f)
System Identification with noise using PSO
(g)
System Identification without noise using DE
(h)
System Identification with noise using DE
(i)
System Identification without noise using CoDE
(j)
System Identification with noise using CoDE
(k)
System Identification without noise using SaDE
(l)
System Identification with noise using SaDE


## 6. Conclusion

In this paper, an improved BSO with differential evolution strategy and new step size method named BSODE is proposed for solving complex optimization problems and training artificial neural networks. Experiments on the 14 widely used benchmark functions of CEC 2005 and the two real-world applications of artificial neural network are carried out to compare the proposed BSODE with five state-of-the-art algorithms in this paper. From these results, we can draw the conclusion that the BSODE significantly improves the performance of the original BSO. Moreover, the BSODE gives good performance on the training of artificial neural networks compared with other algorithms. We expect that BSODE will be used to solve more real-world global optimization problems. Further work includes research into neighbor search to make the algorithm more efficient. In the near future, we also plan to apply the algorithm to large scale optimization domains.



---

*Source: 923698-2015-09-14.xml*