## 5 Classification
### Support Vector Machine  
The support vector machine (SVM) is one of the most effective classification methods that has recently received considerable attention since its introduction by [PERSON] [26]. It is based on the definition of an optimal separating hyperplane (OSH) that separates the training data. It employs a supervised learning approach by labeling the training data with the output class. These are also known as maximum margin classifiers as they simultaneously minimize the empirical risk. The aim of SVM is to determine the optimal hyperplane by maximizing the margin between the separator hyperplane:  
\[\left\{\mathrm{h}\in\mathbb{H}\middle|\left\langle\mathrm{w},\mathrm{h} \right\rangle_{\mathbb{H}}+w_{0}=0\right\} \tag{14}\]  
and the mapped data \(\Phi(\mathbf{s}_{i})\), where \(<\)w, h\(>\) denotes the inner product in space \(\mathbb{H}\), and \(w\) are the hyperplane parameter. Such optimal hyperplane is often considered as the solution of the following Quadratic Programming problem:  
\[\min_{w,w_{0},\xi_{i},...,\xi_{N}}\left(\frac{1}{2}\left\|w\right\|^{2}+C\sum_ {i=1}^{N}\xi_{i}\right) \tag{15}\]  
Subject to  
\[ \begin{gathered} y_{i}(\left\langle\mathrm{w},\Phi(\mathbf{s}_{ i})\right\rangle_{\mathbb{H}}+\mathrm{w}_{0})-1+\xi_{i}\geq 0\;\mathrm{i}=1,...,N\\ \xi_{i}\geq 0\;\;\;\;i=1,...,N\end{gathered}  \tag{16}\]  
where \(N\) is the number of training samples, \(\xi_{i}\) are slack variables, which are introduced to take account of the eventual non-separability of \(\Phi(\mathbf{s}_{i})\) and \(C\) is a positive constant, which controls the trade-off between the slack variable and the size of the margin. The problem in the Equation (15) is solved by using the dual representation and the kernel trick, as:  
\[\max_{\alpha l_{i},...,\alpha l_{N}}\left(\sum_{i=1}^{N}\alpha l_{i}-\frac{1} {2}\sum_{i,j=1}^{N}\alpha l_{i}\alpha l_{j}y_{i}y_{j}K\big{(}\mathbf{s}_{i}, \mathbf{s}_{j}\big{)}\right) \tag{17}\]  
subject to  
\[0\leq\alpha l_{i}\leq C\;i=1,...,N \tag{18}\]\[\sum_{i=1}^{N}\alpha l_{i}y_{i}=0 \tag{19}\]  
where \(K\big{(}s_{i},s_{j}\big{)}=\langle\Phi(s_{i}),\Phi\big{(}s_{j}\big{)}\rangle\) is the kernel function, which is introduced to avoid direct manipulation of the samples in \(\mathbb{H}\), and \(\alpha l_{i}\) are Lagrange multipliers, which can be determined as the solution of a Quadratic Programming problem. In addition to this, SVM kernel is used to map training data implicitly into a higher dimensional feature space. Data vectors which are nearest to the OSH in the higher feature space are called support vectors and contain all information that is required for the classification. Figure 11 depicts the SVM learning approach where a decision boundary separating the two classes is determined with the support vectors that define the margin. In addition, support vectors are considered the key players that define the decision boundary in any SVM. Furthermore, the objective of the SVM is to select the boundary that maximizes the margin, in other words, the boundary with the largest separation between the classes, so that the risk of over-fitting between the classes will be reduced to a minimum. In this work, SVM classification with Gaussian kernel was used. Given that the number of training points is moderate, an SVM classifier with Gaussian kernel is considered as the state of the art choice for a given problem. The Gaussian kernel is defined as:  
\[K(x_{i},x)=\mathrm{e}^{-\|x_{i}-x\|^{2}/2\sigma^{2}} \tag{20}\]  
where \(x_{i}\) belongs to the training data, \(x\) is the support vector and \(\sigma\) is the kernel width, and hyper-parameter of the method, by applying SVM Equation (20) with its specification to the input data (features extraction), the kernel seeks to map the input data to a higher dimensional feature space where the separation between the classes is possible with the output being two classes, e.g., benign and malignant cells.