## 4 Advanced Development of ANNs
### Network Generalization  
Although MFNs can be universal approximators with a sufficiently large number of hidden nodes, excessive number of nodes in the hidden layer may endanger the network to become memorized, which may lead to overfit the input variables [76; 77; 78]. The overfitted networks may fit training data points well but may not be able to well interpolate and extrapolate testing data points. The testing data points can be between the training points or out of the range of the training points. Overfitting is a very important problem in MLPs, and much work has been devoted to preventing overfitting to improve network generalization with techniques such as early stopping, weight decay, and pruning. Figure 6 shows typical error curves of MLP training and testing. With the increase of the hidden nodes, the training error is large at the beginning and then keeps decreasing with a gradual decline of the curve slope. Similarly, the testing error is also larger at the beginning and then keeps decreasing until it reaches a point, h*. Starting from this point, with further increase of the hidden nodes, the testing error increases gradually while the training error continues to decrease. During training, if the minimal point on the testing curve can be found, the overfitting problem may be solved. The training and testing curves with training steps have a similar behavior as shown in Figure 6, which can be used to investigate overfitting, as can happen with too many training steps.  
Based on the explanation of Figure 6, during MLP training, the training error and testing error are checked at every step, and the training process terminates as long as the minimal point of testing error is reached. This is so called early stopping method [20]. Further, a three-set approach was proposed [15]. This approach is to divide a data set into three nonoverlapping subsets: one for training to update network weights, one for testing to terminate training to prevent overfitting, and one for validation to evaluate the trained network. This approach gives a reasonable estimate of the network's generalization ability.  
Adding a penalty term, \(\mathrm{E_{p}}\), to the mean square error of the network outputs in the standard BP algorithm, a new objective function is formulated for MLP training:  
\[\mathrm{E_{w}}=\mathrm{E}+\lambda\mathrm{E_{p}} \tag{13}\]  
where \(\lambda\) is a pre-set constant for the penalty term. The training algorithm using this objective function causes network weights to converge to smaller absolute values than they would in the standard case. This method is called weight decay [16,75].The generalization ability of a network depends on the adjustment of the decay constant, \(\lambda\). With weight decay training, the network can avoid oscillation in the outputs caused by large weights.  
Figure 6: MLP error curves vs. number of hidden nodes.  
Network pruning is another method used to prevent overfitting. This method begins with a fully connected network and the network is made smaller by iteratively eliminating least effective nodes in the hidden layer(s) or interconnections between nodes. The pruned network provides a structure that has a greater capacity for generalization [6, 18, 19].