## 2 Research Methodology
### Cross Validation  
To investigate the problem of underfitting or overfitting resulting from a simple holdout validation, the dataset should be split into a training set, a validation set, and a testing set to judge how the algorithm will perform with new external data. In this case, the training set is used to train the model, the evaluation is performed on the validation set to minimize bias and variance and a final evaluation is performed on the test set to evaluate if the algorithm is either underfitting or overfitting. However, partitioning the available data into three distinct sets will reduce drastically the number of samples used for training the model. As a result, cross validation (CV) was used to evaluate our machine learning models with limited data samples. The test set had to be held out as before, but the validation test was not needed since the cross validation (CV) handled validation by itself within the training set. The main principle behind CV such as \(k\)-fold is to split the training set into \(k\) smaller sets and to train the model on \(k-1\) sets. Validation of the model is performed on the remaining part of the data set to compute the performance measure. This process is repeated \(k\) times so that each \(k\)-fold serves as a validating group once, while the performance measure will be an average of the measure for each \(k\)-fold. In our multi-class application, the f1-macro was adopted as the performance measure. It is defined as the average of the harmonic mean of precision and recall for all classes. The f1-macro was calculated for each \(k\)-fold and averaged to obtain the final performance measure for a fixed set before parameter tuning. The data set was split in to training set (70%) and test set (30%) and the cross validation was carried out within the training set. The process of cross-validation can be looped over to optimize the parameters of each classification method. This strategy to optimize the parameters when training the model is called nested cross validation. In our case, the optimization was performed exhaustively by generating candidates from a grid of parameters relevant to each classifier. Hence, the goal of the grid search was to evaluate all the possible combinations of the parameters within suitable ranges to hold the best combination with regard to the scoring function. SVM, k-NN, RF, AB and GBDT need parameter tuning to achieve their best performances. A limited number of values for each parameter was selected for optimising, as it was unfeasible to try the entire range of values possible. The parameters selected for optimisation were based on literature studies [29; 40; 41]. The parameter value selected for optimisation and the value range are described below:  
* SVM. Three main parameters are selected for optimising the SVM classifier: (i) kernel, which maps the observation into a feature space; (ii) \(C\), which is the regularization parameter that aids in controlling the punishment given to the model for each misclassified point for a given curve; (iii) gamma, which defines how far the influence of a single training example reaches. Three-kernel types were tested: sigmoid, polynomial and radial basis function (RBF). The range for \(C\) and gamma were set from \(10^{-12}\) to \(10^{12}\) using a logarithmic scale. * k-NN. Two main parameters were optimised for the k-NN algorithm: (i) the number of neighbours k with a value range of 1 to 38 in a linear scale and (ii) weight function, namely 'uniform' and 'distance'
* RF. Three parameters were examined for random forest: (i) the number of trees in the forest with a value range \(2^{n}\) [where n = 1 to 9]; (ii) the maximum depth of the tree with a value range of 1-32; and (iii) the minimum number of samples required to split an internal node with a value range of [2 32]. All the vector values were defined using a linear scale. * AB.