# Novel Orthogonal Momentum-Type Particle Swarm Optimization Applied to Solve Large  Parameter Optimization Problems

Authors: Jenn-Long Liu; Chao-Chun Chang
Journal: Journal of Artificial Evolution and Applications
 (2008)

---

## Abstract

This study proposes an orthogonal momentum-type
particle swarm optimization (PSO) that finds good
solutions to global optimization problems using a delta
momentum rule to update the flying velocity of particles
and incorporating a fractional factorial design (FFD) via
several factorial experiments to determine the best
position of particles. The novel combination of the
momentum-type PSO and FFD is termed as the momentum-type PSO
with FFD herein. The momentum-type PSO modifies the
velocity-updating equation of the original Kennedy and
Eberhart PSO, and the FFD incorporates classical
orthogonal arrays into a velocity-updating equation for
analyzing the best factor associated with cognitive
learning and social learning terms. Twelve widely used
large parameter optimization problems were used to
evaluate the performance of the proposed PSO with the
original PSO, momentum-type PSO, and original PSO with
FFD. Experimental results reveal that the proposed
momentum-type PSO with an FFD algorithm efficiently solves
large parameter optimization problems.


---

## Body

## 1. Introduction

Many well-known
representative evolutionary
computation  (EC) techniques, such as genetic algorithms (GAs) [1–3], genetic
programming (GP) [4], evolutionary programming (EP) [5, 6], evolutionary
strategies (ESs) [7, 8], and particle swarm optimization (PSO) [9], have
emerged as efficient computational tools
for solving global optimization problems, especially for large parameter
optimization problems (LPOPs). These EC techniques are sufficiently
robust when coping  with discontinuous, vast multimodal problems or with
noisy search spaces since they do not depend on derivatives of objective
functions and constraints. Among these techniques, PSO is easily
implemented and computationally inexpensive as it has low memory and CPU costs.
The PSO methodology involves creation of a population of particles such that
each particle is capable of adjusting its flying velocity, according to its
flying experience and the best experience of the group. Therefore, each
particle in the PSO can be regarded as a
cooperating agent.The
original PSO algorithm, developed by Kennedy and Eberhart in 1995 [9], was
inspired by the swarm behaviors of flocks of birds and schools of fish. Thus,
the PSO method is a swarm intelligence method [10] for solving global
optimization problems, and it
compares favorably to other evolutionary algorithms [11]. The original PSO is effective in
determining optimal solutions in static environments, but performs poorly in
locating a changing extremum.
In addition, imposing a maximum value Vmax⁡ is necessary to limit the particle velocity.
Shi and Eberhart introduced a parameter called inertia weight (w),
which dampens particles’ velocities over time, allowing a swarm to converge
efficiently and with increased accuracy [12, 13]. Furthermore, Clerc proposed using a constriction factor (K),
which improves the ability of PSO to constrain and control velocities [14].
Eberhart and Shi found that K,
combined with constraints on the maximum allowable velocity vector (Vmax⁡), improved the PSO performance significantly
[15]. Additionally, when utilizing PSO with a constriction factor setting, the Xmax⁡ for each dimension is the best approach. Constriction
factor (K) implements velocity control for effectively
erasing the tendency of some particles to spiral into ever-increasing velocity
oscillations. Parsopoulos and Vrahatis presented an overview of recent
approaches for solving global optimization problems using PSOs [16].This
work presents a novel orthogonal momentum-type PSO algorithm for locating the best position for
each particle efficiently. The momentum-type PSO, proposed in previous work [17], has a
delta momentum rule in the velocity-updating equation, and efficiently
determines the physical motion of
particles more reasonably than other PSO versions. The momentum-type PSO markedly
outperformed the original PSO [9] and modified PSO [13] versions. Furthermore,
this study incorporates
the momentum-type PSO and well-known fractional factorial analysis, which utilize several factorial
experiments, according to classical orthogonal tables, to identify
intelligently the best combination of factors from two-level variables to
enhance algorithmic search efficiency. The novel combination of the
momentum-type PSO and fractional factorial design (FFD) is termed the
momentum-type PSO with FFD herein. Ho et al. [18] and Lin [19] developed
Taguchi orthogonal tables for their GA and PSO for constructing an orthogonal
GA and orthogonal PSO. Their strategies markedly improved the search
ability of their GA and PSO. In 1991, Bhote reported that classical fractional
factorial analysis combined with evolutionary optimization is superior to the Taguchi
method [20]. Consequently,
this study proposes orthogonal PSO using the FFD, rather than Taguchi tables,
for a velocity-updating equation of momentum-type PSO to analyze the best factor related to
cognitive learning and social learning terms with the goal of enhancing
algorithmic efficiency. This
work also developed the original PSO with FFD, which combines the original
Kennedy and Eberhart PSO and classical orthogonal arrays, to enhance
the performance of the original PSO. Performance of the original PSO,
momentum-type PSO, original PSO with FFD, and proposed momentum-type PSO with
FFD were compared using 12 benchmark LPOPs.


## 2. Methodology of the Proposed Particle Swarm Optimization

### 2.1. Description of an Objective Function

The general form of an objective
function with variable vectorx⇀ is expressed as f(x⇀).
The variable x⇀ represents the solution vector with N variables, and is defined as the set {xi,i=1,N}.
In PSO computation, each particle is assigned a fitness value based on the
calculation of objective function. Moreover, the fitness value of a particle
determines the best position of each particle over time and determines which
particle has the best global value in the current swarm.


### 2.2. Momentum-Type Particle Swarm Optimization

The original PSO developed by Kennedy
and Eberhart [9] supposed that the ith particle flies over a hyperspace; its
position and velocity are given by x→i and ν→i,
respectively. Initial particle position and velocity are chosen randomly. At
time step k,
the best position of the current and previous moves for the ith particle
is denoted by pbesti;
the best particle with the best function value in the swarm is denoted by gbest.
Consequently, the next flying velocity and position of particle i at time step k+1 is updated using the following heuristic
equations: (1)ν→ik+1=ν→ik+c1×rand()×(pbesti−x→ik)+c2×rand()×(gbest−x→ik),(2)x→ik+1=x→ik+ν→ik+1,
where c1 and c2 are cognitive and social learning rates,
respectively. These two parameters control the relative importance of memory
(position) of a particle itself and the neighborhood memory, and are both set
to 2.0 [9, 12, 21] for multiplying random values by a mean of 1, such that
agents would “overfly” the target about half the time [9]. Random function rand() is uniformly distributed in the range [0,1].
As in (1), the two random values are generated independently, and the velocity
vector of a particle is updated based on variations on its current position,
its previous best position, and the previous best position of its neighbors.
After particle velocity is updated using (1), its new position is updated by
adding the velocity vector to the current position. Analyses of algorithmic stability
and convergence of the PSO have been examined theoretically by Clerc and
Kennedy [22] and by Trelea [23] who used dynamic system theory.The goal of the momentum-type PSO [17],
which includes a delta term for the velocity vector for detecting particle-velocity
variation, is to express the physical motion of particles more accurately than current
PSO algorithms do. According to [17], the difference between a particle’s
current position x→ik and pbesti (as shown in (1)) represents the first external effect on the particle and
causes it to move, and the difference between a particle’s current position x→ik and gbest,
which represents the second external effect on a particle. The two effects propelling particle i to move can be expressed in delta position forms
as follows:(3)c1×rand()×(pbesti−x→ik)=p×(Δx→ik)1,c2×rand()×(gbest−x→ik)=q×(Δx→ik)2,
where p=c1×rand() and q=c2×rand().
Combining the two terms and using the velocity ν→ik to represent the variation of positions (Δx→ik), the composed velocity V→ik can be obtained as follows:
(4)V→ik=p×(ν→ik)1+q×(ν→ik)2.Sinceν→ik is particle-composed velocity, velocity ν→ik+1 of particle i at next time step k+1 may include a delta form for velocity as Δν→ik, rather than the velocity (ν→ik), as employed in the original PSO, the Shi and
Eberhart PSO, and many other PSOs. Generally, Δν→ik denotes the difference of velocities between
time steps k+1 and k, that is, Δν→ik=ν→ik+1−ν→ik. Thus, the momentum-type form for velocity
updating with parameter β can be written as
(5)ν→ik+1=V→ik+β×Δν→ik,
where β is a positive number (0≤β<1) termed the momentum constant, which controls velocity vector rate of
change. The momentum constant has characteristics that are reminiscent of the
stabilizing effect in backpropagation neural networks [24] and accelerating
convergence in numerical algorithms [25]. Similarly, position x→ik+1 of particle i at time step k+1 can be written as the current position x→ik plus a delta form of the position (Δx→ik):
(6)x→ik+1=x→ik+α×Δx→ik. Parameter α is another momentum constant for adjusting the
rate of change of particle position. Consequently, (5) and (6) correspond to
each other, and both are consistent with the physical behavior for variations
in velocity and position of particle motion. Hence, (5) and (6) can be combined
to form a matrix of the generalized delta rule:
(7)S→ik+1=S→i′k+MΔS→ik,where
(8)S→=[ν→x→],S→′=[V→x→],M=[βα]. Equation
(7) allows each particle the ability of dynamic self-adaptation in the search
space over time, that is, the ith particle can memorize the previous velocity
variation state and automatically adjust the next velocity value during
movement. The delta momentum term in the PSO
algorithm stabilizes searching, and an appropriate value for β=0.1 improves convergence and the optimal solution
[17].  Parameter α was set to 1 such that variable ν could be interpreted as true velocity, that is,
the change between two successive particle positions [23]. Thus (7) satisfies
the zero-velocity variation (Δν→i=0) and zero velocity (ν→i=0) conditions for the particle considered.


### 2.3. Proposed Momentum-Type PSO with Fractional Factorial Design

This
study integrates FFD into the momentum-type PSO to determine the combination of
factors associated with cognitive learning and social learning terms. This study
defined the two states of orthogonal momentum-type PSO as follows:(9)X→i+=x→ki+β×Δν→ik+p×(ν→ik)1,X→i−=x→ki+β×Δν→ik+q×(ν→ik)2, where X→i+ and X→i− indicate two possible new positions related to cognitive
learning and social learning for particle i.
Thus, updating rules for velocity and position using the proposed momentum-type
PSO with FFD can be expressed as(10)ν→ik+1=(X→i+ORX→i−)−x→ik,(11)x→ik+1=(X→i+ORX→i−).
The expression (X→i+ORX→i−) represents the implementation of OR operator via fractional factorial analysis using
variables X→i+ and X→i−.
Therefore, the new flying position of particle i is determined by (11), namely, the best
position of particle i on the next time k+1 can be obtained using FFD.


### 2.4. Fractional Factorial Design (FFD)

To solve the
operator(X→i+ORX→i−) shown in (10) and (11), the FFD used in this
work is based on classical full factorial analysis with two-level, multivariable
orthogonal tables [20]. Applying the fractional factorial analysis to the
operator (X→i+ORX→i−) yields an arrangement of two-level factors
that corresponds to the two possible states of the design variables. In the classical factorial
experiments, the two levels of a design variable are labeled by “−” and “+”.
Hence, Xi− and Xi+ represent the two levels of the design
variable Xi in this work. Table 1 lists the level
distribution of seven factors with the same number of levels “−” and “+” at
each column to retain the balance of a statistically designed experiment.  The detailed steps of the numerical procedure
for the fractional factorial analysis are as follows.Table 1
Orthogonal table with
seven factors.
FactorABABCACBCABCOutputFactor levelX1±X2±X3±X4±X5±X6±X7±Experiment 1−−+−++−f12+−−−−++f23−+−−+−+f34+++−−−−f45−−++−−+f56+−−++−−f67−+−+−+−f78+++++++f8ContributionC1C2C3C4C5C6C7Select level+ or −+ or −+ or −+ or −+ or −+ or −+ or −Best factorX1+ or X1−X2+ or X2−X3+ or X3−X4+ or X4−X5+ or X5−X6+ or X6−X7+ or X7−(1)
First, ifN factors (termed by design variables in the PSO)
each has two levels,
build a table of m rows and m−1 columns. The value m is defined by the integer 2[log⁡2(N+1)].
For example, if seven factors (N=7) are associated with vector X→, then eight
experiments (m=8) are conducted for factorial analysis. As
shown in Table 1, the levels “−” and “+” in columns A, B, and C are assigned to
each cell position according to the classical full factorial principle [20]. In columns AB, AC, BC, and ABC, each level in the cells is determined
from the inner products of columns A and B, columns A and C, columns B and C,
and columns A, B, and C.(2) The first experiment has a factor set toX→={X1−,X2−,X3+,X4−,X5+,X6+,X7−}, obtained by combining the factors A to ABC
with the assigned levels. In the following, its objective function, f1,
is computed, and put it into the first position of column “Output”. Repeat the
computations for the remaining seven experiments with function values f2,f3,…,
and f8;
then, the eight experiments for seven factors are finished.(3) In column A, multiply the function valuesf1,f2,…, and f8 by the corresponding algebraic value −1 for
level “−”, and 1 for level “+”, as listed in Table 1. The effect of factor A on
the level is determined by adding the eight products together. The mathematical
form for each factor i is Ci=∑j=18Ui,j×fj.
Here, Ui,j equals −1 when the cell level is at level “−”,
and equals 1 when it is at level “+”. The summation C1 is placed in the first position in the row
“Contribution” which is associated with factor A. Repeat the multiplication and
summation operations for the remaining six values, C2,C3,…, and C7, and put them in corresponding positions in the
row “Contribution”; the effects of factors B, C,…, and  ABC on the levels are thus obtained.(4) Check
the signs of the seven valuesC1, C2,…, and C7 listed
in the row “Contribution”; if the sign of Ci is negative, then place the symbol “−” in the
row “Selected level” (Table 1). Otherwise, select symbol “+”. The dominant
levels of the seven factors are thus determined.(5) The
best combination ofXi* for the seven factors is determined from the
factors with selected levels, presented in the row “Best factor” (Table 1), and
the best value of the function is obtained by calculating the objective
function f(X→*).
The choice principle for each best factor  Xi* is expressed as follows:
(12)Xi*={Xi−ifsign(Ci)<0Xi+ifsign(Ci)>0.
Thus the new position xik+1 of the particle i at time step k+1 shown in (11) can be set equal to Xi*.


## 2.1. Description of an Objective Function

The general form of an objective
function with variable vectorx⇀ is expressed as f(x⇀).
The variable x⇀ represents the solution vector with N variables, and is defined as the set {xi,i=1,N}.
In PSO computation, each particle is assigned a fitness value based on the
calculation of objective function. Moreover, the fitness value of a particle
determines the best position of each particle over time and determines which
particle has the best global value in the current swarm.


## 2.2. Momentum-Type Particle Swarm Optimization

The original PSO developed by Kennedy
and Eberhart [9] supposed that the ith particle flies over a hyperspace; its
position and velocity are given by x→i and ν→i,
respectively. Initial particle position and velocity are chosen randomly. At
time step k,
the best position of the current and previous moves for the ith particle
is denoted by pbesti;
the best particle with the best function value in the swarm is denoted by gbest.
Consequently, the next flying velocity and position of particle i at time step k+1 is updated using the following heuristic
equations: (1)ν→ik+1=ν→ik+c1×rand()×(pbesti−x→ik)+c2×rand()×(gbest−x→ik),(2)x→ik+1=x→ik+ν→ik+1,
where c1 and c2 are cognitive and social learning rates,
respectively. These two parameters control the relative importance of memory
(position) of a particle itself and the neighborhood memory, and are both set
to 2.0 [9, 12, 21] for multiplying random values by a mean of 1, such that
agents would “overfly” the target about half the time [9]. Random function rand() is uniformly distributed in the range [0,1].
As in (1), the two random values are generated independently, and the velocity
vector of a particle is updated based on variations on its current position,
its previous best position, and the previous best position of its neighbors.
After particle velocity is updated using (1), its new position is updated by
adding the velocity vector to the current position. Analyses of algorithmic stability
and convergence of the PSO have been examined theoretically by Clerc and
Kennedy [22] and by Trelea [23] who used dynamic system theory.The goal of the momentum-type PSO [17],
which includes a delta term for the velocity vector for detecting particle-velocity
variation, is to express the physical motion of particles more accurately than current
PSO algorithms do. According to [17], the difference between a particle’s
current position x→ik and pbesti (as shown in (1)) represents the first external effect on the particle and
causes it to move, and the difference between a particle’s current position x→ik and gbest,
which represents the second external effect on a particle. The two effects propelling particle i to move can be expressed in delta position forms
as follows:(3)c1×rand()×(pbesti−x→ik)=p×(Δx→ik)1,c2×rand()×(gbest−x→ik)=q×(Δx→ik)2,
where p=c1×rand() and q=c2×rand().
Combining the two terms and using the velocity ν→ik to represent the variation of positions (Δx→ik), the composed velocity V→ik can be obtained as follows:
(4)V→ik=p×(ν→ik)1+q×(ν→ik)2.Sinceν→ik is particle-composed velocity, velocity ν→ik+1 of particle i at next time step k+1 may include a delta form for velocity as Δν→ik, rather than the velocity (ν→ik), as employed in the original PSO, the Shi and
Eberhart PSO, and many other PSOs. Generally, Δν→ik denotes the difference of velocities between
time steps k+1 and k, that is, Δν→ik=ν→ik+1−ν→ik. Thus, the momentum-type form for velocity
updating with parameter β can be written as
(5)ν→ik+1=V→ik+β×Δν→ik,
where β is a positive number (0≤β<1) termed the momentum constant, which controls velocity vector rate of
change. The momentum constant has characteristics that are reminiscent of the
stabilizing effect in backpropagation neural networks [24] and accelerating
convergence in numerical algorithms [25]. Similarly, position x→ik+1 of particle i at time step k+1 can be written as the current position x→ik plus a delta form of the position (Δx→ik):
(6)x→ik+1=x→ik+α×Δx→ik. Parameter α is another momentum constant for adjusting the
rate of change of particle position. Consequently, (5) and (6) correspond to
each other, and both are consistent with the physical behavior for variations
in velocity and position of particle motion. Hence, (5) and (6) can be combined
to form a matrix of the generalized delta rule:
(7)S→ik+1=S→i′k+MΔS→ik,where
(8)S→=[ν→x→],S→′=[V→x→],M=[βα]. Equation
(7) allows each particle the ability of dynamic self-adaptation in the search
space over time, that is, the ith particle can memorize the previous velocity
variation state and automatically adjust the next velocity value during
movement. The delta momentum term in the PSO
algorithm stabilizes searching, and an appropriate value for β=0.1 improves convergence and the optimal solution
[17].  Parameter α was set to 1 such that variable ν could be interpreted as true velocity, that is,
the change between two successive particle positions [23]. Thus (7) satisfies
the zero-velocity variation (Δν→i=0) and zero velocity (ν→i=0) conditions for the particle considered.


## 2.3. Proposed Momentum-Type PSO with Fractional Factorial Design

This
study integrates FFD into the momentum-type PSO to determine the combination of
factors associated with cognitive learning and social learning terms. This study
defined the two states of orthogonal momentum-type PSO as follows:(9)X→i+=x→ki+β×Δν→ik+p×(ν→ik)1,X→i−=x→ki+β×Δν→ik+q×(ν→ik)2, where X→i+ and X→i− indicate two possible new positions related to cognitive
learning and social learning for particle i.
Thus, updating rules for velocity and position using the proposed momentum-type
PSO with FFD can be expressed as(10)ν→ik+1=(X→i+ORX→i−)−x→ik,(11)x→ik+1=(X→i+ORX→i−).
The expression (X→i+ORX→i−) represents the implementation of OR operator via fractional factorial analysis using
variables X→i+ and X→i−.
Therefore, the new flying position of particle i is determined by (11), namely, the best
position of particle i on the next time k+1 can be obtained using FFD.


## 2.4. Fractional Factorial Design (FFD)

To solve the
operator(X→i+ORX→i−) shown in (10) and (11), the FFD used in this
work is based on classical full factorial analysis with two-level, multivariable
orthogonal tables [20]. Applying the fractional factorial analysis to the
operator (X→i+ORX→i−) yields an arrangement of two-level factors
that corresponds to the two possible states of the design variables. In the classical factorial
experiments, the two levels of a design variable are labeled by “−” and “+”.
Hence, Xi− and Xi+ represent the two levels of the design
variable Xi in this work. Table 1 lists the level
distribution of seven factors with the same number of levels “−” and “+” at
each column to retain the balance of a statistically designed experiment.  The detailed steps of the numerical procedure
for the fractional factorial analysis are as follows.Table 1
Orthogonal table with
seven factors.
FactorABABCACBCABCOutputFactor levelX1±X2±X3±X4±X5±X6±X7±Experiment 1−−+−++−f12+−−−−++f23−+−−+−+f34+++−−−−f45−−++−−+f56+−−++−−f67−+−+−+−f78+++++++f8ContributionC1C2C3C4C5C6C7Select level+ or −+ or −+ or −+ or −+ or −+ or −+ or −Best factorX1+ or X1−X2+ or X2−X3+ or X3−X4+ or X4−X5+ or X5−X6+ or X6−X7+ or X7−(1)
First, ifN factors (termed by design variables in the PSO)
each has two levels,
build a table of m rows and m−1 columns. The value m is defined by the integer 2[log⁡2(N+1)].
For example, if seven factors (N=7) are associated with vector X→, then eight
experiments (m=8) are conducted for factorial analysis. As
shown in Table 1, the levels “−” and “+” in columns A, B, and C are assigned to
each cell position according to the classical full factorial principle [20]. In columns AB, AC, BC, and ABC, each level in the cells is determined
from the inner products of columns A and B, columns A and C, columns B and C,
and columns A, B, and C.(2) The first experiment has a factor set toX→={X1−,X2−,X3+,X4−,X5+,X6+,X7−}, obtained by combining the factors A to ABC
with the assigned levels. In the following, its objective function, f1,
is computed, and put it into the first position of column “Output”. Repeat the
computations for the remaining seven experiments with function values f2,f3,…,
and f8;
then, the eight experiments for seven factors are finished.(3) In column A, multiply the function valuesf1,f2,…, and f8 by the corresponding algebraic value −1 for
level “−”, and 1 for level “+”, as listed in Table 1. The effect of factor A on
the level is determined by adding the eight products together. The mathematical
form for each factor i is Ci=∑j=18Ui,j×fj.
Here, Ui,j equals −1 when the cell level is at level “−”,
and equals 1 when it is at level “+”. The summation C1 is placed in the first position in the row
“Contribution” which is associated with factor A. Repeat the multiplication and
summation operations for the remaining six values, C2,C3,…, and C7, and put them in corresponding positions in the
row “Contribution”; the effects of factors B, C,…, and  ABC on the levels are thus obtained.(4) Check
the signs of the seven valuesC1, C2,…, and C7 listed
in the row “Contribution”; if the sign of Ci is negative, then place the symbol “−” in the
row “Selected level” (Table 1). Otherwise, select symbol “+”. The dominant
levels of the seven factors are thus determined.(5) The
best combination ofXi* for the seven factors is determined from the
factors with selected levels, presented in the row “Best factor” (Table 1), and
the best value of the function is obtained by calculating the objective
function f(X→*).
The choice principle for each best factor  Xi* is expressed as follows:
(12)Xi*={Xi−ifsign(Ci)<0Xi+ifsign(Ci)>0.
Thus the new position xik+1 of the particle i at time step k+1 shown in (11) can be set equal to Xi*.


## 3. Results and Discussion

### 3.1. Effect of Number of Particles

This work applied FFD to several
factorial experiments to determine the best position of particles. When the
dimension of design variables is high, that is, a large parameter optimization
problem, FFD usually required numerous function evaluations for factorial
analysis. Therefore, few particles for the proposed PSO with FFD should reduce
CPU cost if the good performance of searching of the PSO can be retained. To
determine how many particles are appropriate when using the momentum-type PSO
with FFD and original PSO with FFD algorithms, adequate size ofNparticle is investigated by conducting the following
large parameter optimization problem with low and high dimensions:(13)maximizef(x→)=−∑i=1N[sin⁡(xi)+sin⁡(2xi3)]subjecttoxi∈[3,13].This function is a multimodal problem
and has an analytical optimum of1.216N.  In this computation, dimensions were set at
10 and 100, and the maximum numbers of function evaluation for the two cases
were set at 10000 and 100000, respectively. Moreover, the number of particles
was 5, 10, 15, 20, 25, and 30 to investigate the effects of different swarm
sizes of particles to orthogonal PSOs. Figures 1(a) and 1(b) reveal that the proposed
momentum-type PSO with the FFD algorithm and few particles achieved a favorable
convergence rate and numerical accuracy; nevertheless, many particles resulted
in poor convergence. Using the proposed momentum-type PSO with FFD and few
particles for N=10 and N=100 cases rapidly converges to optimal values
12.1598 and 121.598, respectively. In this study, numerical performance using Nparticle=5 is better than other cases using large number
sizes of particles. Similar results shown in Figures 2(a) and 2(b) are observed
when using the original PSO with the FFD technique. Accordingly, this work set Nparticle=5  for the momentum-type PSO with FFD and the
original PSO with FFD algorithms in the following computations.Effect of number of particles using the momentum-type PSO with FFD for
solving the LPOP at (a)N=5 and (b) N=100 dimensions.
(a)(b)Effect of number of particles using the original PSO with FFD for solving
the LPOP at (a)N=5 and (b) N=100 dimensions.
(a)(b)


### 3.2. Experiments
for 12 Large Parameter Optimization Problems (LPOPs)

Twelve benchmark
LPOPs presented in [18] are evaluated to further examine the performance of the
proposed momentum-type PSO with FFD. This study
sets Nparticle=5 for the original PSO with FFD and the
proposed momentum-type PSO with FFD, and Nparticle=30 for the original PSO and momentum-type PSO.
The number of dimensions for all benchmark LPOPs was set to 100, and the
terminated number of function evaluations for the four PSO algorithms was set
to 100000. Twenty independent runs were conducted for each problem. Algorithmic
performance was assessed by the best solution, averaged best solution, mean
absolute error (MAE, and standard deviation. The MAE was used as the measurement
of error in this work with the following form:(14)MAE=|fopt−favg|N.Table2 lists the 12 objective functions with range
constraints and their global optima [18]. Notably, the objectives of problems
1–3 are set to maximize the objective functions f1–3, and other problems
are set to minimize the objective functions f4–12. The optima of the
first three maximal functions f1,  f2,  and f3 are 121.598,
200, and 185, respectively, and the minimal functions f4–12 are all
zero. Table 3 lists the computational results for the 12 functions at a
dimension of N=100 using the four PSOs. Computational results
reveal that the evaluation results for maximal functions f1, f2, and f3 using the proposed momentum-type PSO with FFD were 121.598,
174.452, and 182.798, respectively. These computational results are better than
those computed by the other three PSOs. Additionally, the algorithmic
performance of the PSOs with FFD is superior to that of PSOs without FFD.
Consequently, the proposed PSO with FFD promoted solution searches more effectively
than the PSO does
without FFD. For the remaining nine minimal function evaluations,  f4–12,
the optima obtained using the proposed momentum-type PSO with FFD were 0.0,
11.946, 0.0, 0.00005, 0.000039, 3781.879, 92.874, 88.0, and 0.000059,
respectively, and the average solutions were 0.0, 32.4641, 0.0, 0.0001, 0.0139,
5676.0005, 181.8753, 88.2285, and 0.1009, respectively. Both the best and
average results are the best when compared with solutions obtained by the other
three PSOs. Additionally, all MAEs and standard deviations are lowest for the
nine optimization functions evaluated by the proposed momentum-type PSO with
FFD over 20 independent runs. From the computational results (Table 3), the
proposed momentum-type PSO with FFD performed well in solving LPOPs. Figures 3(a)–3(l) represent
the convergence histories of the 12 objection functions obtained by the four
PSOs. The proposed momentum-type PSO is superior to the other PSOs in terms of
convergence rate and optimal solution (Figures 3(a)–3(c)). Although
the convergence rate of the original PSO is relatively poor for solving LPOPs,
it can be significantly improved by introducing FFD into the PSO algorithm.
Figures 3(d)–3(l) present
results similar to the maximization cases.Table 2
Twelve objective
functions with range constraints and their optima.
ProblemsObjective functions and rangesOptima1Maximizef1=∑i=1N[sin(xi)+sin(2xi3)];xi∈[3,13]1.21598N2Maximizef2=∑i=1N−1[sin(xi+xi+1)+sin(2xixi+13)];xi∈[3,13]≈ 2N3Maximizef3=∑i=1N[xisin(10πxi)];xi∈[−1,2]1.85N4Minimizef4=∑i=1N[xi+0.5]2;xi∈[−100,100]05Minimizef5=∑i=1N[xi2−10cos(2πxi)+10];xi∈[−5.12,5.12]06Minimizef6=∑i=1N[xi2];xi∈[−5.12,5.12]07Minimizef7=∑i=1N[sin(10xiπ)10xiπ];xi∈[−0.5,0.5]08Minimizef8=−20exp[−0.21N∑i=1Nxi2]−exp[1N∑i=1Ncos(2πxi)]+20+e;xi∈[−30,30]09Minimizef9=418.9828N−∑i=1N(xisin(∣xi∣));xi∈[−500,500]010Minimizef10=∑i=1N−1[100(xi+1−xi2)2+(xi−1)2];xi∈[−5.12,5.12]011Minimizef11=6N+∑i=1N⌊xi⌋;xi∈[−5.12,5.12]012Minimizef12=14000∑i=1Nxi2−∏i=1Ncos(xii)+1;xi∈[−600,600]0Table 3
Results obtained using the original PSO,
momentum-type PSO, original PSO with FFD, and momentum-type PSO with FFD
algorithms to solve the 12 benchmark LPOPs.
f(x)Algorithmsfoptfbestfavg|fopt−favg|MAESD (20 runs)f1Original PSO121.59826.586415.3709106.22711.06235.8322Momentum-type PSO121.5966119.64521.95280.01953.2968Original PSO with FFD121.5730120.62450.97350.00972.7499Momentum-type PSO with FFD121.5980120.37381.22420.01222.2866f2Original PSO20041.332535.9429164.05711.64062.9219Momentum-type PSO147.5408135.863464.13660.64148.0647Original PSO with FFD163.7965150.189249.81080.49816.3295Momentum-type PSO with FFD174.4516166.622833.37720.33386.0346f3Original PSO18541.765636.0755148.92451.48925.6901Momentum-type PSO128.7731116.875468.12460.681211.8977Original PSO with FFD165.4799140.051144.94890.449525.4287Momentum-type PSO with FFD182.7978174.962410.03760.10047.8354f4Original PSO01.3102462.9955462.99554.6300461.6853Momentum-type PSO0.00011.94151.94150.01941.9413Original PSO with FFD3.68289.20729.20720.09215.5244Momentum-type PSO with FFD0.00000.00000.00000.00000.0000f5Original PSO01685.83921774.63151774.631517.746388.7923Momentum-type PSO408.6802565.3514565.351456.5351156.6713Original PSO with FFD35.669576.356776.35670.763640.6872Momentum-type PSO with FFD11.946332.464132.46410.324620.5178f6Original PSO052.4517186.2910186.29101.8629133.8393Momentum-type PSO0.00001.37111.37110.01371.3711Original PSO with FFD0.00000.00000.00000.00000.0000Momentum-type PSO with FFD0.00000.00000.00000.00000.0000f7Original PSO00.00010.00010.00010.00000.0000Momentum-type PSO0.00010.02110.02110.00020.0210Original PSO with FFD0.00010.00010.00010.00000.0000Momentum-type PSO with FFD0.00010.00010.00010.00000.0000f8Original PSO00.40693.95703.95700.03963.5501Momentum-type PSO0.05201.33771.33770.01341.2857Original PSO with FFD0.00000.64760.64760.00650.6476Momentum-type PSO with FFD0.00000.01390.01390.00010.0139f9Original PSO028122.476629843.772929843.7729298.43771721.2964Momentum-type PSO7731.23149897.20219897.202198.97202165.9706Original PSO with FFD5363.77836583.71986583.719865.83721219.9415Momentum-type PSO with FFD3781.87995676.00055676.000556.76001894.1206f10Original PSO0577.886764444.425064444.4250644.444363866.5383Momentum-type PSO373.7911635.0241635.02416.3502261.2330Original PSO with FFD341.03121292.76591292.765912.9277951.7347Momentum-type PSO with FFD92.8743181.8753181.87531.818889.0009f11Original PSO0231.3600319.4241319.42413.194288.0640Momentum-type PSO88.000089.706789.70670.89711.7067Original PSO with FFD88.000089.365389.36530.89371.3653Momentum-type PSO with FFD88.000088.228588.22850.88230.2285f12Original PSO00.2118171.7896171.78961.7179171.5778Momentum-type PSO0.01833.21763.21760.03223.1992Original PSO with FFD0.88767.68157.68150.07686.7939Momentum-type PSO with FFD0.00010.10090.10090.00100.1008Evaluated results obtained using the four PSOs for solving functions (a)f1, (b) f2, (c) f3, (d) f4, (e) f5, (f) f6, (g) f7, (h) f8, (i) f9, (j) f10, (k) f11, and (l) f12.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)


## 3.1. Effect of Number of Particles

This work applied FFD to several
factorial experiments to determine the best position of particles. When the
dimension of design variables is high, that is, a large parameter optimization
problem, FFD usually required numerous function evaluations for factorial
analysis. Therefore, few particles for the proposed PSO with FFD should reduce
CPU cost if the good performance of searching of the PSO can be retained. To
determine how many particles are appropriate when using the momentum-type PSO
with FFD and original PSO with FFD algorithms, adequate size ofNparticle is investigated by conducting the following
large parameter optimization problem with low and high dimensions:(13)maximizef(x→)=−∑i=1N[sin⁡(xi)+sin⁡(2xi3)]subjecttoxi∈[3,13].This function is a multimodal problem
and has an analytical optimum of1.216N.  In this computation, dimensions were set at
10 and 100, and the maximum numbers of function evaluation for the two cases
were set at 10000 and 100000, respectively. Moreover, the number of particles
was 5, 10, 15, 20, 25, and 30 to investigate the effects of different swarm
sizes of particles to orthogonal PSOs. Figures 1(a) and 1(b) reveal that the proposed
momentum-type PSO with the FFD algorithm and few particles achieved a favorable
convergence rate and numerical accuracy; nevertheless, many particles resulted
in poor convergence. Using the proposed momentum-type PSO with FFD and few
particles for N=10 and N=100 cases rapidly converges to optimal values
12.1598 and 121.598, respectively. In this study, numerical performance using Nparticle=5 is better than other cases using large number
sizes of particles. Similar results shown in Figures 2(a) and 2(b) are observed
when using the original PSO with the FFD technique. Accordingly, this work set Nparticle=5  for the momentum-type PSO with FFD and the
original PSO with FFD algorithms in the following computations.Effect of number of particles using the momentum-type PSO with FFD for
solving the LPOP at (a)N=5 and (b) N=100 dimensions.
(a)(b)Effect of number of particles using the original PSO with FFD for solving
the LPOP at (a)N=5 and (b) N=100 dimensions.
(a)(b)


## 3.2. Experiments
for 12 Large Parameter Optimization Problems (LPOPs)

Twelve benchmark
LPOPs presented in [18] are evaluated to further examine the performance of the
proposed momentum-type PSO with FFD. This study
sets Nparticle=5 for the original PSO with FFD and the
proposed momentum-type PSO with FFD, and Nparticle=30 for the original PSO and momentum-type PSO.
The number of dimensions for all benchmark LPOPs was set to 100, and the
terminated number of function evaluations for the four PSO algorithms was set
to 100000. Twenty independent runs were conducted for each problem. Algorithmic
performance was assessed by the best solution, averaged best solution, mean
absolute error (MAE, and standard deviation. The MAE was used as the measurement
of error in this work with the following form:(14)MAE=|fopt−favg|N.Table2 lists the 12 objective functions with range
constraints and their global optima [18]. Notably, the objectives of problems
1–3 are set to maximize the objective functions f1–3, and other problems
are set to minimize the objective functions f4–12. The optima of the
first three maximal functions f1,  f2,  and f3 are 121.598,
200, and 185, respectively, and the minimal functions f4–12 are all
zero. Table 3 lists the computational results for the 12 functions at a
dimension of N=100 using the four PSOs. Computational results
reveal that the evaluation results for maximal functions f1, f2, and f3 using the proposed momentum-type PSO with FFD were 121.598,
174.452, and 182.798, respectively. These computational results are better than
those computed by the other three PSOs. Additionally, the algorithmic
performance of the PSOs with FFD is superior to that of PSOs without FFD.
Consequently, the proposed PSO with FFD promoted solution searches more effectively
than the PSO does
without FFD. For the remaining nine minimal function evaluations,  f4–12,
the optima obtained using the proposed momentum-type PSO with FFD were 0.0,
11.946, 0.0, 0.00005, 0.000039, 3781.879, 92.874, 88.0, and 0.000059,
respectively, and the average solutions were 0.0, 32.4641, 0.0, 0.0001, 0.0139,
5676.0005, 181.8753, 88.2285, and 0.1009, respectively. Both the best and
average results are the best when compared with solutions obtained by the other
three PSOs. Additionally, all MAEs and standard deviations are lowest for the
nine optimization functions evaluated by the proposed momentum-type PSO with
FFD over 20 independent runs. From the computational results (Table 3), the
proposed momentum-type PSO with FFD performed well in solving LPOPs. Figures 3(a)–3(l) represent
the convergence histories of the 12 objection functions obtained by the four
PSOs. The proposed momentum-type PSO is superior to the other PSOs in terms of
convergence rate and optimal solution (Figures 3(a)–3(c)). Although
the convergence rate of the original PSO is relatively poor for solving LPOPs,
it can be significantly improved by introducing FFD into the PSO algorithm.
Figures 3(d)–3(l) present
results similar to the maximization cases.Table 2
Twelve objective
functions with range constraints and their optima.
ProblemsObjective functions and rangesOptima1Maximizef1=∑i=1N[sin(xi)+sin(2xi3)];xi∈[3,13]1.21598N2Maximizef2=∑i=1N−1[sin(xi+xi+1)+sin(2xixi+13)];xi∈[3,13]≈ 2N3Maximizef3=∑i=1N[xisin(10πxi)];xi∈[−1,2]1.85N4Minimizef4=∑i=1N[xi+0.5]2;xi∈[−100,100]05Minimizef5=∑i=1N[xi2−10cos(2πxi)+10];xi∈[−5.12,5.12]06Minimizef6=∑i=1N[xi2];xi∈[−5.12,5.12]07Minimizef7=∑i=1N[sin(10xiπ)10xiπ];xi∈[−0.5,0.5]08Minimizef8=−20exp[−0.21N∑i=1Nxi2]−exp[1N∑i=1Ncos(2πxi)]+20+e;xi∈[−30,30]09Minimizef9=418.9828N−∑i=1N(xisin(∣xi∣));xi∈[−500,500]010Minimizef10=∑i=1N−1[100(xi+1−xi2)2+(xi−1)2];xi∈[−5.12,5.12]011Minimizef11=6N+∑i=1N⌊xi⌋;xi∈[−5.12,5.12]012Minimizef12=14000∑i=1Nxi2−∏i=1Ncos(xii)+1;xi∈[−600,600]0Table 3
Results obtained using the original PSO,
momentum-type PSO, original PSO with FFD, and momentum-type PSO with FFD
algorithms to solve the 12 benchmark LPOPs.
f(x)Algorithmsfoptfbestfavg|fopt−favg|MAESD (20 runs)f1Original PSO121.59826.586415.3709106.22711.06235.8322Momentum-type PSO121.5966119.64521.95280.01953.2968Original PSO with FFD121.5730120.62450.97350.00972.7499Momentum-type PSO with FFD121.5980120.37381.22420.01222.2866f2Original PSO20041.332535.9429164.05711.64062.9219Momentum-type PSO147.5408135.863464.13660.64148.0647Original PSO with FFD163.7965150.189249.81080.49816.3295Momentum-type PSO with FFD174.4516166.622833.37720.33386.0346f3Original PSO18541.765636.0755148.92451.48925.6901Momentum-type PSO128.7731116.875468.12460.681211.8977Original PSO with FFD165.4799140.051144.94890.449525.4287Momentum-type PSO with FFD182.7978174.962410.03760.10047.8354f4Original PSO01.3102462.9955462.99554.6300461.6853Momentum-type PSO0.00011.94151.94150.01941.9413Original PSO with FFD3.68289.20729.20720.09215.5244Momentum-type PSO with FFD0.00000.00000.00000.00000.0000f5Original PSO01685.83921774.63151774.631517.746388.7923Momentum-type PSO408.6802565.3514565.351456.5351156.6713Original PSO with FFD35.669576.356776.35670.763640.6872Momentum-type PSO with FFD11.946332.464132.46410.324620.5178f6Original PSO052.4517186.2910186.29101.8629133.8393Momentum-type PSO0.00001.37111.37110.01371.3711Original PSO with FFD0.00000.00000.00000.00000.0000Momentum-type PSO with FFD0.00000.00000.00000.00000.0000f7Original PSO00.00010.00010.00010.00000.0000Momentum-type PSO0.00010.02110.02110.00020.0210Original PSO with FFD0.00010.00010.00010.00000.0000Momentum-type PSO with FFD0.00010.00010.00010.00000.0000f8Original PSO00.40693.95703.95700.03963.5501Momentum-type PSO0.05201.33771.33770.01341.2857Original PSO with FFD0.00000.64760.64760.00650.6476Momentum-type PSO with FFD0.00000.01390.01390.00010.0139f9Original PSO028122.476629843.772929843.7729298.43771721.2964Momentum-type PSO7731.23149897.20219897.202198.97202165.9706Original PSO with FFD5363.77836583.71986583.719865.83721219.9415Momentum-type PSO with FFD3781.87995676.00055676.000556.76001894.1206f10Original PSO0577.886764444.425064444.4250644.444363866.5383Momentum-type PSO373.7911635.0241635.02416.3502261.2330Original PSO with FFD341.03121292.76591292.765912.9277951.7347Momentum-type PSO with FFD92.8743181.8753181.87531.818889.0009f11Original PSO0231.3600319.4241319.42413.194288.0640Momentum-type PSO88.000089.706789.70670.89711.7067Original PSO with FFD88.000089.365389.36530.89371.3653Momentum-type PSO with FFD88.000088.228588.22850.88230.2285f12Original PSO00.2118171.7896171.78961.7179171.5778Momentum-type PSO0.01833.21763.21760.03223.1992Original PSO with FFD0.88767.68157.68150.07686.7939Momentum-type PSO with FFD0.00010.10090.10090.00100.1008Evaluated results obtained using the four PSOs for solving functions (a)f1, (b) f2, (c) f3, (d) f4, (e) f5, (f) f6, (g) f7, (h) f8, (i) f9, (j) f10, (k) f11, and (l) f12.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)


## 4. Conclusion

This study examined the performance of the proposed momentum-type PSO
with the FFD algorithm, which uses the delta momentum rule to update particle velocity
and FFD to locate a particle’s best location for handling LPOPs. Using the
momentum-type PSO, each particle can adjust its velocity vector according to
its previous velocity change rate. The
new location of a particle is then determined by analyzing the best combination
of cognitive and social learning terms via several factorial experiments based
on the FFD. Moreover, this work modified the original PSO by incorporating with
FFD, which also employs the classical orthogonal tables, and improved
significantly the efficiency of the original
Kennedy and Eberhart PSO. Through 12 benchmark function assessments, the proposed momentum-type
PSO algorithm outperformed the original PSO, momentum-type PSO and original PSO
with FFD in terms of solution accuracy, numerical error and standard deviation.
This work also determined that algorithmic performance of the proposed PSO with
FFD is superior to that without FFD. Numerical results show that the proposed momentum-type
PSO with FFD algorithm is efficient and is a promising method for solving
LPOPs.



---

*Source: 827401-2008-03-31.xml*