## 4 Modeling
### Machine-Learning Techniques and Best Metrics
#### 4.2.1 Classification
\[K(x_{i},x_{j})=e^{-\frac{\|x-x_{j}\|^{2}}{\gamma^{2}}}\] (6) The letter \(\gamma\) represents the bandwidth, and it is a parameter of the method. When an SVM does not use a kernel, it is said to be an SVM with a linear kernel. Whether or not a linear kernel is used, the C parameter is present and must be specified. Adapting SVM to multi-class problems is done using the 1 vs. 1 strategy, whereby a problem of n classes becomes (n\(-\)1)! binary problems. These problems result from confronting each class with all the rest. For each binary problem, an SVM is then trained, and the final prediction comes from the majority vote of all those SVMs. Two implementations of SVM were used in the experiments:
* LibSVM [39]: used to build Radial Basis Function kernel SVMs. * LibLinear [40]: used to compute linear kernel SVMs. For LibLinear, the C parameter was optimized, and in the case of LibSVM, the \(\gamma\) parameter was also optimized. In all cases, cross validation was used on the training set. The aim of both optimizations was to maximize the F-macro, using the same modification of the WEKA Multisearch package described above for the Multilayer Perceptron. 5. Bagging-based methods [41] in which several decision trees are created. These trees are different because they are trained by resampling the training set. The final prediction in these ensembles is the one obtained by the majority vote of each of these trees. The trees used for the ensemble are commonly referred to as base classifiers. The Bagging variants used in the experiment were: 1. Bagging using C4.5 trees as base classifiers 2. Bagging using Random Balance of C4.5 trees [42]. Random Balance changes the number of instances of each class before they are used by each C4.5. Rather than balancing the number of instances of each class, this technique assigns a random number of instances to each. To do so, it both resamples the training set when it needs less instances, and it also creates synthetic instances as and when it needs. To create these synthetic instances, it uses SMOTE [27]. Random Balance is oriented towards unbalanced multiclass datasets, so it is _a priori_ an approach that fits the nature of the problem under analysis very well. 3. Random Forest [43] can be considered a Bagging technique in which the base classifiers are \"Random Trees\". In these decision trees, each time a branching node is to be built, the possible attributes to be considered are randomly restricted. In WEKA, Random Trees are implemented as a modification of REP-Trees (i.e., Reduced Error Pruning trees), which have a fast building process. 4. Rotation Forest [44]. In this method, before building each tree, and once the training set has been resampled, the features are grouped randomly. These groups are disjointed from each other; and the union of all the groups contains all the attributes. All groups have the same number of attributes n, except perhaps one of the groups, when the dataset number of attributes is not divisible by n. In each group, the PCA projection is computed, taking only a number of principal components (i.e., n in most cases) to keep the size of the groups unchanged. In the experiments, Rotation Forest of C 4.5 was used, as was Rotation Forest of Random Forest, because its training time is significantly shorter with the same number of trees. All Bagging configurations have been computed using 100 trees. For this reason, Rotation Forest of Random Forest calculates 10 Random Forests, each containing 10 trees. 6. Finally, two Boosting based ensembles were included for classification: 1. AdaBoost.M1 [45], because it is the most popular Boosting ensemble for classification. In this ensemble each base classifier is derived from the previous one, so that it gives more weight to the instances that the base classifier of the immediately preceding iteration has incorrectly classified. Unlike Bagging, the final prediction is not made by a majority vote of the base classifiers, but by a vote weighted by the individual error of each one.