## 3 Classification Description
### Support vector machine classification  
Another popular network model that uses Gaussian functions are support vector machines (SVM). SVM generally solve the two class problem with a network structure that is similar to the RBF structure in Figure 4. Since there are only two classes, SVM only have one output. The main difference between the two networks relies on the underlying approach and in the way they are trained. Where the RBF network places Gaussian kernels on the cluster centres formed by the features in the feature space, the SVM network places the Gaussian kernels on those samples that define the boundary between two classes. These points are found by creating a separating hyperplane inside a higher dimensional feature space. A reason to consider SVM is that they tend to show strong generalisation performance [22]. In the case of SVM, the interpretation of the first layer is that it projects the data to a feature space where the two classes can be separated by a hyperplane. The position of the hyperplane is decided by those points that lie within a certain margin between the two classes (the support vectors). The projected feature space is allowed (or preferred) to have a higher dimension, as there is no penalty in the form of computational drawbacks or the 'curse of dimensionality' since the actual mapping is never performed. All necessary calculations in the projected feature space are evaluated through inner products, which can be done with a kernel function. In our case this was the Gaussian function (3) that was also used for the RBF network. The training stage for SVM both defines the number of nodes in the hidden layer and calculates the output layer weights at the same time. There several approaches to train the network in the case of non-separable classes. First we looked at \(C\)-SVM, which aims at solving the following optimisation problem [27]:  
\[\underset{\mathbf{w},\xi,b}{\text{min}}\ ||\mathbf{w}||^{2}+C\sum_{i=1}^{N}\xi_{i} \tag{4}\]  
under the constraint that the training data should classify correctly, and where \(\mathbf{w}\) is the direction of the normal vector on the separating plane, \(N\) the number of samples, \(\xi\) are the slack vectors which are non-zero for the samples that lie within the margin, and \(C\) is the pre-defined penalty on these points. This problem has a straightforward optimal solution \(\mathbf{w}_{o}\) using Lagrange multipliers [27], expressed as follows\[\mathbf{w}_{o}=\sum_{i=1}^{N}\alpha_{i}t_{i}\phi(\mathbf{s}_{i}) \tag{5}\]  
where \(t_{i}\) are the targets of mapped inputs \(\mathbf{s}_{i}\). Only a few of the Lagrange multipliers \(\alpha_{i}\) will be non-zero, and those define both the first layer support vectors (\(\mathbf{s}_{i}\)) and the second layer weights. It should be noted that \(\mathbf{w_{o}}\) is never actually evaluated as the classification function given by  
\[f(\mathbf{s})\;=\;\mathbf{w}_{o}^{T}\phi(\mathbf{s})+b_{o}\;=\;\sum_{i\;\in\; \mathrm{s}\
u}\alpha_{i}t_{i}\phi(\mathbf{s}_{i})\cdot\phi(\mathbf{s})+b_{o} \tag{6}\]  
evaluates the inner product directly though the kernel function. A second training method that trains a SVM network regulating classification errors is \(\
u\)-SVM [15]. The optimisation problem that is solved is then given by  
\[\underset{\mathbf{w},\xi,b,\rho}{\text{min}}\;\;\frac{1}{2}||\mathbf{w}||^{2} -\
u\rho+\frac{1}{N}\sum_{i=1}^{N}\xi_{i} \tag{7}\]  
under the constraint of correct classification, and where \(\rho\) is the margin in the mapped feature space and is left as a free variable. The fixed constant \(\
u\) gives some control over both the error rate on the training set (\(P_{e}\), the ratio between misclassifications and number of training patterns) and the number of support vectors (\(N_{s}\)) from the optimisation through the relationships [22] :  
\[P_{e}\leq\
u;\qquad\text{and}\qquad N\
u\leq N_{s}. \tag{8}\]  
This can be a useful property as the number of support vectors play a large role on the speed of the network.