## 4 Advanced Development of ANNs
### SVMs  
SVMs were proposed by [PERSON] [85]. Based on the description of [PERSON] [32] and [PERSON] and [PERSON] [33], SVMs are a set of supervised generalized linear classifiers that have often been found to provide higher classification accuracies than widely used MLP classifiers. The growing interest in SVMs is shown by their successful application in many areas such as automobile [86], time series analysis [87, 88], medicine [89], pattern recognition [90], geophysics [91, 92], computational biology [93], and control engineering [94]. The interest resulted from SVMs' intrinsic effectiveness with respect to traditional classifiers. This effectiveness results in high classification accuracies and very good generalization capabilities. In architecture design, SVMs requires limited effort to configure few control parameters compared with traditional classifiers such as MLPs. Also, based on the universal approximation capability of their standard MFN counterparts, the approximation capability of SVMs was investigated [51]. It was shown that an SVM with polynomial kernel of degree \(n\)-1, which is trained on a training set of size \(n\) can approximate the n training points up to any accuracy. A SVM performs classification through mapping input vectors into a higher-dimensional space and constructing a hyperplane that optimally separates the data in the higher-dimensional space. A SVM model using sigmoid kernel function is equivalent to a two-layer perceptron neural network. However, the operation of the SVM is different. In training, a SVM implements a simple linear mapping or linear classifier together with a prior fixed nonlinear mapping in order to make data separable. This implementation allows training to not suffer from the problem of local minima, and to focus on function optimization with respect to its generalization ability. These advantages often lead to better results for SVMs compared to MFNs. The goal of SVM modeling is to find the optimal hyperplane that separates clusters of vectors, a set of features, such that data points with one class of the variable are on one side of the plane and ones with the other class are on the other side of the plane. The vectors near the hyperplane are the support vectors. Let's look at the 2-dimensional case as an example. Assume that for classification, the data are from a categorical variable with two classes, and there are two prediction variables, X and Y, with continuous values. The data points using the value of X on the X axis and Y on the Y axis may be plotted as Figure 9.  
In the plot it can be seen what classes each (X,Y) point belongs to. In this example, the data points with one class are in the lower left corner of the plot and the data points with the other classes are in the upper right corner. The SVM analysis is find a one-dimensional hyperplane, i.e. a line, to separate the data points based on their target classes. Obviously, there are an infinite number of possible such lines. Two of them are shown in the plot. The task of SVM now is to determine which line is optimal. For this optimization two parallel (dashed) lines are constructed, one on each side of the separating line. The two lines are pushed up against the two sets of data points belonged to two classes. They label the distance between the separating line and the closest vectors to the line. The two lines are used to calculate the margin, which is defined as the distance between them. The SVM analysis is to find the line (hyperplane in general) that makes the margin between the support vectors maximized. In Figure 9, the line A is superior to the line B. In this sense, SVM classifiers are also known as maximum margin classifiers. Mathematically, suppose having a training data pairs {(x\({}_{i}\), c\({}_{i}\)) \(|\) = 1, 2,..., N} with x\({}_{i}\) \(\in\) R\({}^{n}\) and c\({}_{i}\)\(\in\) {1, -1}, indicate what class the data point x\({}_{i}\) belongs to.