## 4 Advanced Development of ANNs
### Standard BP Enhancement  
In standard BP training, it is critical to select a learning rate, \(\eta\), to let the process converge to the true global minimum of the mean square error of the network outputs with a rather rapid speed. A BP training with a too small learning rate will have a noticeable slow progress. One with a too large learning rate will speed up significantly. However, this may result in oscillations around relatively poor solutions. Evidence shows that the use of a momentum term in the BP algorithm can be helpful in speeding the convergence and avoiding local minima ([[http://www.cs.bham.ac.uk/~pxt/NC/](http://www.cs.bham.ac.uk/~pxt/NC/)]([http://www.cs.bham.ac.uk/~pxt/NC/](http://www.cs.bham.ac.uk/~pxt/NC/)) ASSIGNMENT/MICHAEL/momentum.html). The momentum term is defined as a fraction of the previous weight change \(\alpha\Delta\mathrm{w}^{\prime}\), and added to equation (2) from the standard BP algorithm [7]:  
\[\Delta\mathrm{w}=-\eta\
abla\mathrm{E}(\mathrm{w})+\alpha\Delta\mathrm{w}^{\prime} \tag{12}\]  
where \(\alpha\) is taken \(0\leq\alpha\leq 0.9\). Figure 5: Topological neighborhoods with the progress of training as feature maps are formed. In this way, in BP training, the momentum is used to stabilize the weight change using a combination of the gradient decreasing term with a fraction of the previous weight change. With the concept of momentum in BP training, the method has been further developed. [PERSON] _et al._[9] developed an adaptive momentum algorithm which can update the momentum coefficient automatically in every iteration step. The result of this adaptive process is equivalent to adding a momentum term to the standard BP algorithm. The momentum coefficient is updated automatically in every iteration. Numerical simulations show that the adaptive momentum algorithm can eliminate possible divergent oscillations during the initial training, and can also accelerate the learning process and result in a lower error when the final convergence is reached. [PERSON] and [PERSON] [10] presented an analysis of fuzzy adaption of training parameters (learning rate and momentum) to accelerate BP learning in MFNs. There are other studies on BP training acceleration and local minima. [PERSON] _et al._[75] investigated the efficiency of the training processes of MFNs using the gradient descent method in the standard BP algorithm and Levenberg-Marquardt method in backpropagation. It was found that in the case of low epoch training (below several thousand epochs) using the gradient descent algorithm, the Levenberg-Marquardt algorithm was less efficient, and in the case of high epoch training (above several thousand epochs) using the gradient descent algorithm, the Levenberg-Marquardt algorithm was more efficient. [PERSON] and [PERSON] [11] described properties of various parameters of sigmoid activation function mathematically with their influence on the speed of convergence of the BP training algorithm for MFNs. [PERSON] _et al._[12] proposed an improved BP algorithm intended to avoid the local minima problem caused by neuron saturation in the hidden layer. Each training pattern has its own activation functions of neurons in the hidden layer. When the network outputs have not received their desired signals, the activation functions are adapted so as to prevent neurons in the hidden layer from saturating. [PERSON] _et al._[13] proposed a modified error function with two terms. By adding one term to the conventional error function, the modified error function can harmonize the update of weights connected to the hidden layer and those connected to the output layer. Thus, it can avoid the local minima problem caused by update disharmony between weights connected to the hidden layer and the output layer. Simulations on some benchmark problems and a real classification task have been performed to test the validity of the modified error function.