## 3 Methodology
### Modelling
#### 3.4.2 Support Vector Machines  
Support vector machines (SVMs) are primarily constructed for multiclass classification, although they can perform binary separation. The idea of SVMs is that a classification problem with \(N\) number of input features can be solved by finding a hyperplane of dimension \(N-1\). The hyperplane separates the \(N\)-dimensional space in \(N\) parts where the data points in the same subspace also belong to the same class. The equation for the separating hyperplane will have several solutions [34]. For the sake of simplicity, consider a linear SVM where \(N=3\), then the hyperplane is a line. The line can be moved sideways between its two closest points to separate and even be tilted in new angles and still separate training data points of the \(N\)-classes into their own spaces. A poorly chosen hyperplane out of the alternatives may make the performance on test data suffer, although the training performance is the same. A similar problem will be found in higher dimensions and non-linear settings as well. To obtain a good model, a good hyperplane must be found. One such hyperplane is the maximum margin hyperplane. The maximum margin hyperplane is the maximum distance to the data points closest to the hyperplane, thus a hyperplane with the maximal possible margin. The data points on the margin to the hyperplane are called \"support vectors\" since they support the placement of the hyperplane. The maximal margin hyperplane is only dependent on the support vectors for its positioning. If the training set is changed by adding or removing data points, it will not affect the classifier unless the set of support vectors is altered. However, it is not satisfactory that the classifier can be fundamentally changed by adding just one training sample. Moreover, the scenario that there is no perfectly separating hyperplane needs a solution. A solution for both problems is to introduce a soft margin. The soft margin introduces an error tolerance of the model, which allows some of the training data points to be on the wrong side of the margin, or even the wrong side of the hyperplane. The constraint added is that the total errors may sum up to a specific constant but no more. Data points within the margin will also be considered as support vectors. A linear separating hyperplane will follow Equation (16):  
\[0=\beta_{0}+\beta_{1}X_{1}+\beta_{2}X_{2}+\ldots+\beta_{N}X_{n} \tag{16}\]  
where \(\beta_{i}\ \in\{\beta_{i},\ldots,\beta_{N}a\) are the parameters to find by training. The corresponding classification function is explained in Equation (17):  
\[f(x)=\beta_{0}+\beta_{1}x_{1}+\beta_{2}x_{2}+\ldots+\beta_{N}x_{n} \tag{17}\]  
Here, \(x_{i}\) are the features of the sample to classify. The class of a new data point is determined by whether \(f(x)\) has a value above or below zero. The linear classification function of the SVM utilises the inner products of the observations. Therefore, the classification function can be rewritten as described in Equation (18):  
\[f(x)=\beta_{0}+\sum_{i=1}^{N}\alpha_{i}K(x,x_{i}) \tag{18}\]  
where \(\alpha_{i}\) are the parameters found by the training, and \(K(x,x_{i})\) is the inner product between observations (Equation (19)):  
\[K\big{(}x_{i},x^{\prime}_{i}\big{)}=\sum_{j=1}^{p}x_{i}x^{\prime}_{i} \tag{19}\]  
SVM has primarily been constructed for multiclass classification. The idea of SVM is that a classification problem with \(N\) number of input features can be solved by finding a hyperplane of dimension \(N-1\). SVM classification is an optimisation problem. Linear discriminant analysis (LDA) and kernel functions are two analytical solutions used for optimisation. We utilised kernel methods for SVM to transform a linear classifier into a non-linear classifier. In the linear classifier, the inner product is called the kernel function, or Kernel for short. The kernel of a classifier quantifies the similarity of two observations. To separate classes that have non-linear boundaries, the hyperplane must be described by a non-linear equation.