## 4 Modeling
### Machine-Learning Techniques and Best Metrics  
#### 4.2.1 Classification  
The metric usually used for classification problems is the percentage of test instances in which the trained model correctly predicts the class. However, in unbalanced problems this metric will not properly evaluate the models that are obtained, since, for example, a classifier that always predicts mode 1 would always achieve a 74% success rate. There are several metrics for unbalanced classification, but many of them are targeted at two-class problems. For multi-class problems, it is usual to use the Matthews correlation coefficient (i.e., MCC), and the F-measure in both its micro version (i.e., F-micro) and its macro version (i.e., F-macro). The Matthews Correlation Coefficient, or MCC, is a state-of-the-art metric for unbalanced problems that has a multi-class variant [31]. To compute it, the confusion matrix must be previously calculated. This is a square matrix in which the number of rows and columns correspond to the number of classes in the problem. Therefore, the elements in the diagonal of the matrix represent the cases in which the classifier predicts the correct class. Once the matrix is obtained, the MCC can be calculated as:  
\[MCC=\frac{\sum_{k}\sum_{l}\sum_{m}C_{k,k}C_{l,m}-C_{k,l}C_{m,k}}{\sqrt{\sum_{k} (\sum_{l}C_{k,l})(\sum_{k^{\prime}|k^{\prime}\
eq k}\sum_{l^{\prime}}C_{k^{ \prime},l^{\prime}}))\sqrt{(\sum_{k}(\sum_{l}C_{l,k}))(\sum_{k^{\prime}|k^{ \prime}\
eq k}\sum_{l^{\prime}}C_{l^{\prime},l^{\prime}})}}} \tag{1}\]  
The F-Measure metric, for the binary case was calculated from the Precision (Pr) and Recall (Re) metrics. \[P_{r} :=\frac{TP}{TP+FP} \tag{2}\] \[R_{e} :=\frac{TP}{TP+FN} \tag{3}\]  
where:  
* TP (i.e., true positives): number of times the classifier correctly predicts the minority class. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**State** & **No. of Instances** & **\%** & **Mode** & **No. of Instances** & **\%** \\ \hline
1 & 18.288 & 34.75\% & 1 & 38,938 & 74.00\% \\
2 & 29.124 & 55.40\% & 2 & 7,034 & 13.37\% \\
3 & 468 & 0.89\% & 3 & 5,463 & 10.38\% \\
4 & 4.712 & 8.95\% & 4 & 700 & 1.33\% \\  & & & 5 & 487 & 0.93\% \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Distribution of class values in the two classification problems. * FP (i.e., false positives): number of times the classifier incorrectly predicts instances of the majority class as instances of the minority class. * FN (i.e., false negatives): number of times the classifier is wrong when predicting minority class instances as majority class instances. Precision is therefore the ratio of the number of instances of the minority class to the instances predicted as belonging to the minority class; while recall is the ratio of the number of instances predicted as belonging to the minority class to the size of that class. The F-measure metric combines both measures according to the following equation:  
\[F_{measure}=2*\frac{P_{r}*R_{e}}{P_{r}+R_{e}} \tag{4}\]  
Hence, the F-Measure is the harmonic mean of both magnitudes. By expanding Equation (4), another expression can be found for the F-measure. \[F_{measure}=\frac{2*TP}{(2*TP+FP+FN)} \tag{5}\]  
There are two variants to fit this metric to the multi-class case: F-Macro average and F-Micro average. The F-Macro computation sets up a binary classification problem for each class, where each class acts as a minority class versus an artificial class consisting of the union of all the other classes (i.e., OVA = One vs. All). Then, for each of these problems the F-measure is calculated, and by averaging all these F-measures the F-Macro is finally obtained. In the case of the F-Micro, we obtain for each class the values of TP, FP, and FN. All the TPs are added up to obtain a global TP. A global FP and FN could be obtained in the same way. Then, Equation (5) is applied with the three global values, and the F-measure then obtained is the F-Micro. [PERSON] et al. [32] reported some situations where a division by zero can be achieved in the F-Macro calculation using cross validation.