# An Advanced Conjugate Gradient Training Algorithm Based on a Modified Secant Equation

Authors: Ioannis E. Livieris; Panagiotis Pintelas
Journal: ISRN Artificial Intelligence
 (2012)

---

## Abstract

Conjugate gradient methods constitute excellent neural network training methods characterized by their simplicity, numerical efficiency, and their very low memory requirements. In this paper, we propose a conjugate gradient neural network training algorithm which guarantees sufficient descent using any line search, avoiding thereby the usually inefficient restarts. Moreover, it achieves a high-order accuracy in approximating the second-order curvature information of the error surface by utilizing the modified secant condition proposed by Li et al. (2007). Under mild conditions, we establish that the proposed method is globally convergent for general functions under the strong Wolfe conditions. Experimental results provide evidence that our proposed method is preferable and in general superior to the classical conjugate gradient methods and has a potential to significantly enhance the computational efficiency and robustness of the training process.


---

## Body

## 1. Introduction

Learning systems, such as multilayer feedforward neural networks (FNN), are parallel computational models comprised of densely interconnected, adaptive processing units, characterized by an inherent propensity for learning from experience and also discovering new knowledge. Due to their excellent capability of self-learning and self-adapting, they have been successfully applied in many areas of artificial intelligence [1–5] and are often found to be more efficient and accurate than other classification techniques [6]. The operation of a FNN is usually based on the following equations:
(1)netjl=∑i=1Nl-1wijl-1,lyil-1+bjl,yjl=f(netjl),
where netjl is the sum of its weighted inputs for the jth node in the lth layer (j=1,…,Nl), wijl-1,l are the weights from the ith neuron at the (l-1) layer to the jth neuron at the lth layer, bjl is the bias of the jth neuron at the lth layer, yil is the output of the jth neuron that belongs to the lth layer, and f(netjl) is the jth neuron activation function.The problem oftraining a neural network is to iteratively adjust its weights, in order to globally minimize a measure of difference between the actual output of the network and the desired output for all examples of the training set [7]. More mathematically, the training process can be formulated as the minimization of the error function E(w), defined by the sum of square differences between the actual output of the FNN, denoted by yj,pL and the desired output, denoted by tj,p, relative to the appeared output, namely,(2)E(w)=∑p=1P∑j=1NL(yj,pL-tj,p)2,
where w∈ℝn is the vector network weights and P represents the number of patterns used in the training set.Conjugate gradient methods are probably the most famous iterative methods for efficiently training neural networks due to their simplicity, numerical efficiency, and their very low memory requirements. These methods generate a sequence of weights{wk} using the iterative formula(3)wk+1=wk+ηkdk,k=0,1,…,
where k is the current iteration usually called epoch, w0∈ℝn is a given initial point, ηk>0 is the learning rate, and dk is a descent search direction defined by(4)dk={-g0,ifk=0,-gk+βkdk-1,otherwise,
where gk is the gradient of E at wk and βk is a scalar. In the literature, there have been proposed several choices for βk which give rise to distinct conjugate gradient methods. The most well-known conjugate gradient methods include the Fletcher-Reeves (FR) method [8], the Hestenes-Stiefel (HS) method [9], and the Polak-Ribière (PR) method [10]. The update parameters of these methods are, respectively, specified as follows:(5)βkHS=gkTyk-1yk-1Tdk-1,βkFR=‖gk‖2‖gk-1‖2,βkPR=gkTyk-1‖gk-1‖2,
where sk-1=xk-xk-1, yk-1=gk-gk-1 and ∥·∥ denotes the Euclidean norm.The PR method behaves like the HS method in practical computation and it is generally believed to be one of the most efficient conjugate gradient methods. However, despite the practical advantages of this method, it has the major drawback of not being globally convergent for general functions and as a result it may be trapped and cycle infinitely without presenting any substantial progress [11]. For rectifying the convergence failure of the PR method, Gilbert and Nocedal [12], motivated by Powell’s work [13], proposed to restrict the update parameter βk of being nonnegative, namely, βkPR+=max{βkPR,0}. The authors conducted an elegant analysis of this conjugate gradient method (PR+) and established that it is globally convergent under strong assumptions. Moreover, although that the PR method and the PR+ method usually perform better than the other conjugate gradient methods, they cannot guarantee to generate descent directions, hence restarts are employed in order to guarantee convergence. Nevertheless, there is also a worry with restart algorithms that their restarts may be triggered too often; thus degrading the overall efficiency and robustness of the minimization process [14].During the last decade, much effort has been devoted to develop new conjugate gradient methods which are not only globally convergent for general functions but also computationally superior to classical methods and are classified in two classes. The first class utilizes second-order information to accelerate conjugate gradient methods by utilizing new secant equations (see [15–18]). Sample works include the nonlinear conjugate gradient methods proposed by Zhang et al. [19–21] which are based on MBFGS secant equation [15]. Ford et al. [22] proposed a multistep conjugate gradient method that is based on the multistep quasi-Newton methods proposed in [16, 17]. Recently, Yabe and Takano [23] and Li et al. [18] proposed conjugate gradient methods which are based on modified secant equation using both the gradient and function values with higher orders of accuracy in the approximation of the curvature. Under proper conditions, these methods are globally convergent and sometimes their numerical performance is superior to classical conjugate gradient methods. However, these methods do not ensure to generate descent directions; therefore the descent condition is usually assumed in their analysis and implementations.The second class aims at developing conjugate gradient methods which generate descent directions, in order to avoid the usually inefficient restarts. On the basis of this idea, Zhang et al. [20, 24–26] modified the search direction in order to ensure sufficient descent, that is, dkTgk=-∥gk∥2, independent of the performed line search. Independently, Hager and Zhang [27] modified the parameter βk and proposed a new descent conjugate gradient method, called the CG-DESCENT method. More analytically, they proposed a modification of the Hestenes-Stiefel formula βkHS in the following way:(6)βkHZ=βkHS-2‖yk-1‖2(dk-1Tyk-1)2gkTdk-1.
Along this line, Yuan [28] based on [12, 27, 29], proposed a modified PR method, that is,(7)βkDPR+=βkPR-min⁡{βkPR,C‖yk-1‖2‖gk-1‖4gkTdk-1},
where C is a parameter which essentially controls the relative weight between conjugacy and descent and in case C>1/4 then the above formula satisfies gkTdk≤-(1-1/4C)∥gk∥2. An important feature of this method is that it is globally convergent for general functions. Recently, Livieris et al. [30–32] motivated by the previous works presented some descent conjugate gradient training algorithms providing some promising results. Based on their numerical experiments, the authors concluded that the sufficient descent property led to a significant improvement of the training process.In this paper, we proposed a new conjugate gradient training algorithm which has both characteristics of the previous presented classes. Our method ensures sufficient descent independent of the accuracy of the line search, avoiding thereby the usually inefficient restarts. Moreover, it achieves a high-order accuracy in approximating the second-order curvature information of the error surface by utilizing the modified secant condition proposed in [18]. Under mild conditions, we establish the global convergence of our proposed method.The remainder of this paper is organized as follows. In Section2, we present our proposed conjugate gradient training algorithm and in Section 3, we present its global convergence analysis. The experimental results are reported in Section 4 using the performance profiles of Dolan and Morè [33]. Finally, Section 5 presents our concluding remarks.


## 2. Modified Polak-Ribière+ Conjugate Gradient Algorithm

Firstly, we recall that for quasi-Newton methods, an approximation matrixBk-1 to the Hessian ∇2E(wk-1) of a nonlinear function E is updated so that a new matrix Bk satisfies the following secant condition:(8)Bksk-1=yk-1.
Obviously, only two gradients are exploited in the secant equation (8), while the function values available are neglected. Recently, Li et al. [18] proposed a conjugate gradient method based on the modified secant condition(9)Bksk-1=ỹk-1,ỹk-1=yk-1+max⁡{θk-1,0}‖sk-1‖2sk-1,
where θk-1 is defined by(10)θk-1=2(Ek-1-Ek)+(gk+gk-1)Tsk-1,
and Ek denotes E(wk). The authors proved that this new secant equation (9) is superior to the classical one (8) in the sense that ỹk-1 better approximates ∇2E(wk)sk-1 than yk-1 (see [18]).Motivated by the theoretical advantages of this modified secant condition (9), we propose a modification of formula (7), in the following way:(11)βkMPR+=gkTỹk-1‖gk-1‖2-min⁡{gkTỹk-1‖gk-1‖2,C‖ỹk-1‖2‖gk-1‖4gkTdk-1},
with C>1/4. It is easy to see from (4) and (11) that our proposed formula βkMPR+ satisfies the sufficient descent condition(12)gkTdk≤-(1-14C)‖gk‖2,
independent of the line search used.At this point, we present a high level description of our proposed algorithm, called modified Polak-Ribière+ conjugate gradient algorithm (MPR+-CG).Algorithm 1 (modified Polak-Ribière+ conjugate gradient algorithm).
Step 1.
Initiatew0, 0<σ1<σ2<1, EG and kMAX; set k=0.Step 2.
Calculate the error function valueEk and its gradient gk.Step 3.
If(Ek<EG), return w*=wk and E*=Ek.Step 4.
If(gk=0), return “Error goal not met”.Step 5.
Compute the descent directiondk using (4) and (11).Step 6.
Compute the learning rateηk using the strong Wolfe line search conditions
(13)E(wk+αkdk)-E(wk)≤σ1αkgkTdk,(14)|g(wk+αkdk)Tdk|≤σ2|gkTdk|.Step 7.
Update the weights(15)wk+1=wk+ηkdk
and set k=k+1.Step 8.
If(k>kMAX) return “error goal not met”, else go to Step 2.


## 3. Global Convergence Analysis

In order to establish the global convergence result for our proposed method, we will impose the following assumptions on the error functionE.Assumption 1.
The level setℒ={w∈ℝn∣E(w)≤E(w0)} is bounded.Assumption 2.
In some neighborhood𝒩∈ℒ, E is differentiable and its gradient g is Lipschitz continuous, namely, there exists a positive constant L>0 such that
(16)‖g(w)-g(w̃)‖≤L‖w-w̃‖,and∀w,w̃∈N.Since{E(wk)} is a decreasing sequence, it is clear that the sequence {wk} is contained in ℒ. In addition, it follows directly from Assumptions 1 and 2 that there exist positive constraints B and M, such that
(17)‖w-w̃‖≤B,∀w,w̃∈L,(18)‖g(w)‖≤M,∀w∈L.
Furthermore, notice that since the error function E is bounded below in ℝn by zero, it is differentiable and its gradient is Lipschitz continuous [34]. Assumptions 1 and 2 always hold.The following lemma is very useful for the global convergence analysis.Lemma 2 (see [18]).
Suppose that Assumptions1 and 2 hold and the line search satisfies the strong Wolfe line search conditions (13) and (14). For θk and ỹk defined in (10) and (9), respectively, one has
(19)|θk|≤L‖sk‖2,‖ỹk‖≤2L‖sk‖.
Subsequently, we will establish the global convergence of Algorithm MPR+-CG for general functions. Firstly, we present a lemma that Algorithm MPR+-CG prevents the inefficient behavior of the jamming phenomenon [35] from occurring. This property is similar to but slightly different from Property(*), which was derived by Gilbert and Nocedal [12].Lemma 3.
Suppose that Assumptions1 and 2 hold. Let {wk} and {dk} be generated by Algorithm MPR+-CG, if there exists a positive constant μ>0 such that
(20)‖gk‖≥μ,∀k≥0,
then there exist constants b>1 and λ>0 such that
(21)|βkMPR+|≤b,(22)‖sk-1‖≤λ⟹|βkMPR+|≤1b.Proof.
Utilizing Lemma2 together with Assumption 2 and relations (12), (14), (17), (18), (20) we have
(23)|βkMPR+|≤|gkTỹk-1|‖gk-1‖2+C‖ỹk-1‖2‖gk-1‖4|gkTdk-1|≤‖gk‖‖ỹk-1‖‖gk-1‖2+C‖ỹk-1‖2σ2|gk-1Tdk-1|‖gk-1‖4≤2ML‖sk-1‖‖gk-1‖2+C4L2‖sk-1‖2‖gk-1‖4σ2(1-14C)‖gk-1‖2≤(2ML+4L2BCσ2(1-1/4C)μ2)‖sk-1‖≜D‖sk-1‖.
Therefore, by setting b:=max{2,2DB} and λ:=1/Db, we have relations (21) and (22) hold. The proof is completed.Subsequently, we present a lemma which shows that, asymptotically, the search directionswk change slowly. This lemma corresponds to Lemma 4.1 of Gilbert and Nocedal [12] and the proof is exactly the same as that of Lemma 4.1 in [12], thus we omit it.Lemma 4.
Suppose that Assumptions1 and 2 hold. Let {wk} and {dk} be generated by Algorithm MPR+-CG, if there exists a positive constant μ>0 such that (21) holds; then dk≠0 and
(24)∑k≥1‖uk-uk-1‖2<∞,
where uk=dk/∥dk∥.Next, by making use of Lemmas3 and 4, we establish the global convergence theorem for Algorithm MPR+-CG under the strong Wolfe line search.Theorem 5.
Suppose that Assumptions1 and 2 hold. If {wk} is obtained by Algorithm MPR+-CG where the line search satisfies the strong Wolfe line search conditions (13) and (14), then one has
(25)lim⁡k→∞inf⁡‖gk‖=0.Proof.
We proceed by contraction. Suppose that there exists a positive constantμ>0 such that for all k≥0(26)‖gk‖≥μ.
The proof is divided in the following two steps.Step I
A bound on the stepsk. Let Δ be a positive integer, chosen large enough that
(27)Δ≥4BD,
where B and D are defined in (17) and (23), respectively. For any l>k≥k0 with l-k≤Δ, following the same proof as the case II of Theorem 3.2 in [27], we get
(28)∑j=kl-1‖sj‖<2B.Step II
A bound on the search directions ofdl. It follows from the definition of dk in (4) together with (18) and (23), we obtain
(29)‖dl‖2≤‖gl‖2+|βkMPR+|2‖dl-1‖2≤M2+D2‖sl-1‖2‖dl-1‖2.Now, the remaining argument is standard in the same way as case III in Theorem 3.2 in [27], thus we omit it. This completes the proof.


## 4. Experimental Results

In this section, we will present experimental results in order to evaluate the performance of our proposed conjugate gradient algorithm MPR+-CG in five famous classification problems acquired by the UCI Repository of Machine Learning Databases [36]: the iris problem, the diabetes problem, the sonar problem, the yeast problem, and the Escherichia coli problem.The implementation code was written in Matlab 6.5 on a Pentium IV computer (2.4 MHz, 512 Mbyte RAM) running Windows XP operating system based on the SCG code of Birgin and Martínez [37]. All methods are implemented with the line search proposed in CONMIN [38] which employs various polynomial interpolation schemes and safeguards in satisfying the strong Wolfe line search conditions. The heuristic parameters were set as σ1=10-4 and σ2=0.5 as in [30, 39]. All networks have received the same sequence of input patterns and the initial weights were generated using the Nguyen-Widrow method [40]. For evaluating classification accuracy we, have used the standard procedure called k-fold cross-validation [41]. The results have been averaged over 500 simulations.


### 4.1. Training Performance

The cumulative total for a performance metric over all simulations does not seem to be too informative, since a small number of simulations can tend to dominate these results. For this reason, we use the performance profiles proposed by Dolan and Morè [33] to present perhaps the most complete information in terms of robustness, efficiency, and solution quality. The performance profile plots the fraction P of simulations for which any given method is within a factor τ of the best training method. The horizontal axis of each plot shows the percentage of the simulations for which a method is the fastest (efficiency), while the vertical axis gives the percentage of the simulations that the neural networks were successfully trained by each method (robustness). The reported performance profiles have been created using the Libopt environment [42] for measuring the efficiency and the robustness of our method in terms of computational time (CPU time) and function/gradient evaluations (FE/GE). The curves in the following figures have the following meaning.(i)
“PR’’ stands for the Polak-Ribière conjugate gradient method.(ii)
“PR+’’ stands for the Polak-Ribière+ conjugate gradient method.(iii)
“MPR+’’ stands for Algorithm MPR+-CG.


#### 4.1.1. Iris Classification Problem

This benchmark is perhaps the most best known to be found in the pattern-recognition literature [36]. The data set contains 3 classes of 50 instances each, where each class refers to a type of iris plant. The network architectures constitute of 1 hidden layer with 7 neurons and an output layer of 3 neurons. The training goal was set to EG≤0.01 within the limit of 1000 epochs and all networks were tested using 10-fold cross-validation [30].Figure1 presents the performance profiles for the iris classification problem, regarding both performance metrics. MPR+ illustrates the best performance in terms of efficiency and robustness, significantly outperforming the classical training methods PR and PR+. Furthermore, the performance profiles show that MPR+ is the only method reporting an excellent (100%) probability of being the optimal training method.Log10 scaled performance profiles for the iris classification problem.
(a)
Performance based on CPU time(b)
Performance based on FE/GE


#### 4.1.2. Diabetes Classification Problem

The aim of this real-world classification task is to decide whether a Pima Indian female is diabetes positive or not. The data of this benchmark consists of 768 different patterns, each of them having 8 features of real continuous values and a class label (diabetes positive or not). We have used neural networks with 2 hidden layers of 4 neurons each and an output layer of 2 neurons [43]. The training goal was set to EG<0.14 within the limit of 2000 epochs and all networks were tested using 10-fold cross-validation [44].Figure2 illustrates the performance profiles for the diabetes classification problem, investigating the efficiency and robustness of each training method. Clearly, our proposed method MPR+ significantly outperforms the conjugate gradient methods PR and PR+ since the curves of the former lie above the curves of the latter, regarding both performance metrics. More analytically, the performance profiles show that the probability of MPR+ to successfully train a neural network within a factor 3.41 of the best solver is 100%, in contrast with PR and PR+ which have probability 84.3% and 85%, respectively.Log10 scaled performance profiles for the diabetes classification problem.
(a)
Performance based on CPU time(b)
Performance based on FE/GE


#### 4.1.3. Sonar Classification Problem

This is the dataset used by Gorman and Sejnowski [45] in their study of the classification of sonar signals using a neural network. The dataset contains signals obtained from a variety of different aspect angles, spanning 90 degrees for the cylinder and 180 degrees for the rock. The network architecture for this problem constitutes of 1 hidden layer of 24 neurons and an output layer of 2 neurons [45]. The training goal was set to EG=0.1 within the limit of 1000 epochs and all networks were tested using 3-fold cross-validation [45].In Figure3 are presented the performance profiles for the sonar classification problem, relative to both performance metrics. Our proposed conjugate gradient method MPR+ presents the highest probability of being the optimal training method. Furthermore, MPR+ significantly outperforms PR and is slightly more robust than PR+, regarding both performance metrics.Log10 scaled performance profiles for the sonar classification problem.
(a)
Performance based on CPU time(b)
Performance based on FE/GE


#### 4.1.4. Yeast Classification Problem

This problem is based on a drastically imbalanced dataset and concerns the determination of the cellular localization of the yeast proteins into ten localization sites.Saccharomyces cerevisiae (yeast) is the simplest Eukaryotic organism. The network architecture for this classification problem consists of 1 hidden layer of 16 neurons and an output layer of 10 neurons [46]. The training goal was set to EG<0.05 within the limit of 2000 epochs and all networks were tested using 10-fold cross validation [47].Figure4 presents the performance profiles for the yeast classification problem, regarding both performance metrics. The interpretation in Figure 4 highlights that our proposed conjugate gradient method MPR+ is the only method exhibiting an excellent (100%) probability of successful training. Moreover, it is worth noticing that PR and PR+ report very poor performance exhibiting 0% and 5% probability of successfully training, respectively, in contrast with our proposed method MPR+ which has successfully trained all neural networks.Log10 scaled performance profiles for the yeast classification problem.
(a)
Performance based on CPU time(b)
Performance based on FE/GE


#### 4.1.5.Escherichia coli Classification Problem

This problem is based on a drastically imbalanced data set of 336 patterns and concerns the classification of theE. coli protein localization patterns into eight localization sites. E. coli, being a prokaryotic gram-negative bacterium, is an important component of the biosphere. Three major and distinctive types of proteins are characterized in E. coli: enzymes, transporters, and regulators. The largest number of genes encoding enzymes (34%) (this should include all the cytoplasm proteins) is followed by the genes for transport functions and the genes for regulatory process (11.5%) [48]. The network architectures constitute of 1 hidden layer with 16 neurons and an output layer of 8 neurons [46]. The training goal was set to EG≤0.02 within the limit of 2000 epochs and all neural networks were tested using 4-fold cross-validation [47].In Figure5 are presented the performance profiles for the Escherichia coli classification problem. Similar observations can be made with the previous benchmarks. More specifically, MPR+ significantly outperforms the classical training methods PR and PR+, since the curves of the former lie above the curves of the latter, regarding both performance metrics. Moreover the performance profiles show that the probability of MPR+ is the only method reporting excellent (100%) probability of being the optimal training method.Log10 scaled performance profiles for the Escherichia coli classification problem.
(a)
Performance based on CPU time(b)
Performance based on FE/GE


### 4.2. Generalization Performance

In Table1 are summarized the generalization results of PR, PR+, and MPR+ conjugate gradient methods, measured by the percentage of testing patterns that were classified correctly in the presented classification problems. Each row reports the average performance in percentage for each problem and the best conjugate gradient method for a problem is illustrated in boldface. Moreover, “−” means that the method reported 0% training success.Table 1
Generalization results of conjugate gradient methods PR, PR+, and MPR+.
Classification problemsMethodIrisDiabetesSonarYeastE.coliPR95.0%76.1%75.9%—95.8%PR+95.3%76.1%75.6%92.0%96.0%MPR+98.1%76.4%75.9%92.5%96.0%The interpretation on Table1 illustrates that MPR+ is an excellent generalizer since it manages to have the highest generalization performance, outperforming the classical training methods PR and PR+ in all classification problems.


## 4.1. Training Performance

The cumulative total for a performance metric over all simulations does not seem to be too informative, since a small number of simulations can tend to dominate these results. For this reason, we use the performance profiles proposed by Dolan and Morè [33] to present perhaps the most complete information in terms of robustness, efficiency, and solution quality. The performance profile plots the fraction P of simulations for which any given method is within a factor τ of the best training method. The horizontal axis of each plot shows the percentage of the simulations for which a method is the fastest (efficiency), while the vertical axis gives the percentage of the simulations that the neural networks were successfully trained by each method (robustness). The reported performance profiles have been created using the Libopt environment [42] for measuring the efficiency and the robustness of our method in terms of computational time (CPU time) and function/gradient evaluations (FE/GE). The curves in the following figures have the following meaning.(i)
“PR’’ stands for the Polak-Ribière conjugate gradient method.(ii)
“PR+’’ stands for the Polak-Ribière+ conjugate gradient method.(iii)
“MPR+’’ stands for Algorithm MPR+-CG.


### 4.1.1. Iris Classification Problem

This benchmark is perhaps the most best known to be found in the pattern-recognition literature [36]. The data set contains 3 classes of 50 instances each, where each class refers to a type of iris plant. The network architectures constitute of 1 hidden layer with 7 neurons and an output layer of 3 neurons. The training goal was set to EG≤0.01 within the limit of 1000 epochs and all networks were tested using 10-fold cross-validation [30].Figure1 presents the performance profiles for the iris classification problem, regarding both performance metrics. MPR+ illustrates the best performance in terms of efficiency and robustness, significantly outperforming the classical training methods PR and PR+. Furthermore, the performance profiles show that MPR+ is the only method reporting an excellent (100%) probability of being the optimal training method.Log10 scaled performance profiles for the iris classification problem.
(a)
Performance based on CPU time(b)
Performance based on FE/GE


### 4.1.2. Diabetes Classification Problem

The aim of this real-world classification task is to decide whether a Pima Indian female is diabetes positive or not. The data of this benchmark consists of 768 different patterns, each of them having 8 features of real continuous values and a class label (diabetes positive or not). We have used neural networks with 2 hidden layers of 4 neurons each and an output layer of 2 neurons [43]. The training goal was set to EG<0.14 within the limit of 2000 epochs and all networks were tested using 10-fold cross-validation [44].Figure2 illustrates the performance profiles for the diabetes classification problem, investigating the efficiency and robustness of each training method. Clearly, our proposed method MPR+ significantly outperforms the conjugate gradient methods PR and PR+ since the curves of the former lie above the curves of the latter, regarding both performance metrics. More analytically, the performance profiles show that the probability of MPR+ to successfully train a neural network within a factor 3.41 of the best solver is 100%, in contrast with PR and PR+ which have probability 84.3% and 85%, respectively.Log10 scaled performance profiles for the diabetes classification problem.
(a)
Performance based on CPU time(b)
Performance based on FE/GE


### 4.1.3. Sonar Classification Problem

This is the dataset used by Gorman and Sejnowski [45] in their study of the classification of sonar signals using a neural network. The dataset contains signals obtained from a variety of different aspect angles, spanning 90 degrees for the cylinder and 180 degrees for the rock. The network architecture for this problem constitutes of 1 hidden layer of 24 neurons and an output layer of 2 neurons [45]. The training goal was set to EG=0.1 within the limit of 1000 epochs and all networks were tested using 3-fold cross-validation [45].In Figure3 are presented the performance profiles for the sonar classification problem, relative to both performance metrics. Our proposed conjugate gradient method MPR+ presents the highest probability of being the optimal training method. Furthermore, MPR+ significantly outperforms PR and is slightly more robust than PR+, regarding both performance metrics.Log10 scaled performance profiles for the sonar classification problem.
(a)
Performance based on CPU time(b)
Performance based on FE/GE


### 4.1.4. Yeast Classification Problem

This problem is based on a drastically imbalanced dataset and concerns the determination of the cellular localization of the yeast proteins into ten localization sites.Saccharomyces cerevisiae (yeast) is the simplest Eukaryotic organism. The network architecture for this classification problem consists of 1 hidden layer of 16 neurons and an output layer of 10 neurons [46]. The training goal was set to EG<0.05 within the limit of 2000 epochs and all networks were tested using 10-fold cross validation [47].Figure4 presents the performance profiles for the yeast classification problem, regarding both performance metrics. The interpretation in Figure 4 highlights that our proposed conjugate gradient method MPR+ is the only method exhibiting an excellent (100%) probability of successful training. Moreover, it is worth noticing that PR and PR+ report very poor performance exhibiting 0% and 5% probability of successfully training, respectively, in contrast with our proposed method MPR+ which has successfully trained all neural networks.Log10 scaled performance profiles for the yeast classification problem.
(a)
Performance based on CPU time(b)
Performance based on FE/GE


### 4.1.5.Escherichia coli Classification Problem

This problem is based on a drastically imbalanced data set of 336 patterns and concerns the classification of theE. coli protein localization patterns into eight localization sites. E. coli, being a prokaryotic gram-negative bacterium, is an important component of the biosphere. Three major and distinctive types of proteins are characterized in E. coli: enzymes, transporters, and regulators. The largest number of genes encoding enzymes (34%) (this should include all the cytoplasm proteins) is followed by the genes for transport functions and the genes for regulatory process (11.5%) [48]. The network architectures constitute of 1 hidden layer with 16 neurons and an output layer of 8 neurons [46]. The training goal was set to EG≤0.02 within the limit of 2000 epochs and all neural networks were tested using 4-fold cross-validation [47].In Figure5 are presented the performance profiles for the Escherichia coli classification problem. Similar observations can be made with the previous benchmarks. More specifically, MPR+ significantly outperforms the classical training methods PR and PR+, since the curves of the former lie above the curves of the latter, regarding both performance metrics. Moreover the performance profiles show that the probability of MPR+ is the only method reporting excellent (100%) probability of being the optimal training method.Log10 scaled performance profiles for the Escherichia coli classification problem.
(a)
Performance based on CPU time(b)
Performance based on FE/GE


## 4.1.1. Iris Classification Problem

This benchmark is perhaps the most best known to be found in the pattern-recognition literature [36]. The data set contains 3 classes of 50 instances each, where each class refers to a type of iris plant. The network architectures constitute of 1 hidden layer with 7 neurons and an output layer of 3 neurons. The training goal was set to EG≤0.01 within the limit of 1000 epochs and all networks were tested using 10-fold cross-validation [30].Figure1 presents the performance profiles for the iris classification problem, regarding both performance metrics. MPR+ illustrates the best performance in terms of efficiency and robustness, significantly outperforming the classical training methods PR and PR+. Furthermore, the performance profiles show that MPR+ is the only method reporting an excellent (100%) probability of being the optimal training method.Log10 scaled performance profiles for the iris classification problem.
(a)
Performance based on CPU time(b)
Performance based on FE/GE


## 4.1.2. Diabetes Classification Problem

The aim of this real-world classification task is to decide whether a Pima Indian female is diabetes positive or not. The data of this benchmark consists of 768 different patterns, each of them having 8 features of real continuous values and a class label (diabetes positive or not). We have used neural networks with 2 hidden layers of 4 neurons each and an output layer of 2 neurons [43]. The training goal was set to EG<0.14 within the limit of 2000 epochs and all networks were tested using 10-fold cross-validation [44].Figure2 illustrates the performance profiles for the diabetes classification problem, investigating the efficiency and robustness of each training method. Clearly, our proposed method MPR+ significantly outperforms the conjugate gradient methods PR and PR+ since the curves of the former lie above the curves of the latter, regarding both performance metrics. More analytically, the performance profiles show that the probability of MPR+ to successfully train a neural network within a factor 3.41 of the best solver is 100%, in contrast with PR and PR+ which have probability 84.3% and 85%, respectively.Log10 scaled performance profiles for the diabetes classification problem.
(a)
Performance based on CPU time(b)
Performance based on FE/GE


## 4.1.3. Sonar Classification Problem

This is the dataset used by Gorman and Sejnowski [45] in their study of the classification of sonar signals using a neural network. The dataset contains signals obtained from a variety of different aspect angles, spanning 90 degrees for the cylinder and 180 degrees for the rock. The network architecture for this problem constitutes of 1 hidden layer of 24 neurons and an output layer of 2 neurons [45]. The training goal was set to EG=0.1 within the limit of 1000 epochs and all networks were tested using 3-fold cross-validation [45].In Figure3 are presented the performance profiles for the sonar classification problem, relative to both performance metrics. Our proposed conjugate gradient method MPR+ presents the highest probability of being the optimal training method. Furthermore, MPR+ significantly outperforms PR and is slightly more robust than PR+, regarding both performance metrics.Log10 scaled performance profiles for the sonar classification problem.
(a)
Performance based on CPU time(b)
Performance based on FE/GE


## 4.1.4. Yeast Classification Problem

This problem is based on a drastically imbalanced dataset and concerns the determination of the cellular localization of the yeast proteins into ten localization sites.Saccharomyces cerevisiae (yeast) is the simplest Eukaryotic organism. The network architecture for this classification problem consists of 1 hidden layer of 16 neurons and an output layer of 10 neurons [46]. The training goal was set to EG<0.05 within the limit of 2000 epochs and all networks were tested using 10-fold cross validation [47].Figure4 presents the performance profiles for the yeast classification problem, regarding both performance metrics. The interpretation in Figure 4 highlights that our proposed conjugate gradient method MPR+ is the only method exhibiting an excellent (100%) probability of successful training. Moreover, it is worth noticing that PR and PR+ report very poor performance exhibiting 0% and 5% probability of successfully training, respectively, in contrast with our proposed method MPR+ which has successfully trained all neural networks.Log10 scaled performance profiles for the yeast classification problem.
(a)
Performance based on CPU time(b)
Performance based on FE/GE


## 4.1.5.Escherichia coli Classification Problem

This problem is based on a drastically imbalanced data set of 336 patterns and concerns the classification of theE. coli protein localization patterns into eight localization sites. E. coli, being a prokaryotic gram-negative bacterium, is an important component of the biosphere. Three major and distinctive types of proteins are characterized in E. coli: enzymes, transporters, and regulators. The largest number of genes encoding enzymes (34%) (this should include all the cytoplasm proteins) is followed by the genes for transport functions and the genes for regulatory process (11.5%) [48]. The network architectures constitute of 1 hidden layer with 16 neurons and an output layer of 8 neurons [46]. The training goal was set to EG≤0.02 within the limit of 2000 epochs and all neural networks were tested using 4-fold cross-validation [47].In Figure5 are presented the performance profiles for the Escherichia coli classification problem. Similar observations can be made with the previous benchmarks. More specifically, MPR+ significantly outperforms the classical training methods PR and PR+, since the curves of the former lie above the curves of the latter, regarding both performance metrics. Moreover the performance profiles show that the probability of MPR+ is the only method reporting excellent (100%) probability of being the optimal training method.Log10 scaled performance profiles for the Escherichia coli classification problem.
(a)
Performance based on CPU time(b)
Performance based on FE/GE


## 4.2. Generalization Performance

In Table1 are summarized the generalization results of PR, PR+, and MPR+ conjugate gradient methods, measured by the percentage of testing patterns that were classified correctly in the presented classification problems. Each row reports the average performance in percentage for each problem and the best conjugate gradient method for a problem is illustrated in boldface. Moreover, “−” means that the method reported 0% training success.Table 1
Generalization results of conjugate gradient methods PR, PR+, and MPR+.
Classification problemsMethodIrisDiabetesSonarYeastE.coliPR95.0%76.1%75.9%—95.8%PR+95.3%76.1%75.6%92.0%96.0%MPR+98.1%76.4%75.9%92.5%96.0%The interpretation on Table1 illustrates that MPR+ is an excellent generalizer since it manages to have the highest generalization performance, outperforming the classical training methods PR and PR+ in all classification problems.


## 5. Conclusions

In this paper, we proposed a conjugate gradient method for efficiently training neural networks. An attractive property of our proposed method is that it ensures sufficient descent, avoiding thereby the usually inefficient restarts. Furthermore, it achieves a high-order accuracy in approximating the second-order curvature information of the error surface by utilizing the modified secant equation proposed in [18]. Under mild conditions, we established that our proposed method is globally convergent. Based on our numerical experiments, we concluded that our proposed method outperforms classical conjugate gradient training methods and has a potential to significantly enhance the computational efficiency and robustness of the training process.



---

*Source: 486361-2011-12-08.xml*