## 7 Experiments
### Experimental Results with Features Selection  
A further experiment was held by executing a preliminary features selection procedure in order to reduce the dimensionality of the numeric vectors and make the prediction model focus on the most informative linguistic features. The features selection was performed by means of the well known recursive features elimination (RFE) algorithm [71]. The RFE algorithm recursively fits the model, ranks the features according to a measure of their relevance to the classification process, and finally removes the weakest one. For the tree-based models RF and GBDT, the features importance is calculated as the _Gini importance_ index [72], while for the rest of the models we used the well known _permutation features importance_ technique [73]. Moreover, to find the optimal number of features, RFE was cross-validated in order to score feature subsets of different sizes. For each model, the number of features selected is provided in Table 4. Interestingly, the number of selected features does not vary greatly across the different models and it ranges from 48 (kNNU) to 62 (NB). Hence, more than half of the totality of the features is removed by RFE. Furthermore, the set of selected features is quite stable among the different models. Hence, all the classification models considered in this work were analyzed using the reduced sets of features identified by the features selection procedure. The accuracy scores of the cross validation process are provided in Table 5, together with the improvement of accuracy with respect to the experiment analyzed in Section 7.1 and a symbol that indicates if the improvement is statistically significant or not (the symbol \"=\" indicates no difference with the previous experiment, while \"\(\blacktriangle\)\" and \"\(\blacktriangledown\)\" indicate that the new execution, respectively, significantly outperforms or is significantly outperformed by the previous one). The statistical test considered is the Mann-Whitney U test with a significance threshold of 0.05. The models in Table 5 are ordered (from left to right) by accuracy. RF, as in the previous example, is the model showing the larger accuracy, 74.1%, which improves the previous experiment by 1.6%. This improvement is not statistically significant, though the p-value of the statistical test, 0.08, is close to the significance threshold. Overall, all the models show an improvement of accuracy with respect to the previous experiment. The improvement is statistically significant for MLP1, kNNU, kNND, QDA, and NB. It is interesting to observe the large improvement for the two kNN schemes observed using the features space with reduced dimensionality. By recalling that kNN works by computing distances in the features space, the improvement in accuracy can be interpreted as the fact that the removed features were, in some sense, spatially misleading. Finally, we analyze the selected features by considering the best performing model RF. Figure 4a shows the distribution of the selected features within each feature category (see Section 5). As we can see, morpho-syntactic, syntactic, and lexical features constitute the categories with most features represented in the final set. Furthermore, in Figure 4b, we provide the normalized Gini importance \begin{table}
\begin{tabular}{l c c c c c c c c c c} \hline \hline  & **RF** & **SVM** & **MLP3** & **MLP2** & **MLP1** & **GBDT** & **kNNU** & **kNND** & **QDA** & **NB** \\ \hline
**Number of sel. features** & 58 & 54 & 53 & 51 & 53 & 59 & 48 & 50 & 60 & 62 \\
**Percentage of sel. features** & 41.7\% & 38.8\% & 38.1\% & 36.7\% & 38.1\% & 42.4\% & 34.5\% & 36.0\% & 43.2\% & 44.6\% \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Amount of selected features per model.