## 3 Interpretability Analysis in the Prediction of Cardiovascular Diseases
### Performance Evaluation Methods  
To measure the balance between performance and interpretability, the set of classification techniques was compared using a different number of features. A total of 100 runs were carried out for each classifier. Each run was a 4-fold cross-validation process that measured the average performance of the algorithms when they had all the features. Then, each was run with 66% of the best features (cp, thal, ca, oldpeak, thalach, exang, slope, and sex) according to the rank value in Table 2 and finally with 33% of the best features (cp, thal, ca, and oldpeak). The performance of the algorithms was compared when different amounts of attributes were used. Figure 3 shows a diagram of the process. Remember that we were not looking for the best classifier, we wanted to compare the effect of using different amounts of features on the accuracy and interpretability of the models. Figures 4 and 5 are shown boxplots: in blue and labeled as 100 the results when all features were used, in green when 66% and red when 33% of the most important features were used. Accuracy and AUC were selected as quality measures. We compared 36 classifiers, most of which were implementations of the R caret library [40] and a few belong to  
\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c c} \hline \hline  & **Chi\({}^{2}\)** & **GainR** & **OneR** & **SU** & **IG** & **Rel** & **KT** & **PCF** & **Rimp** & **Rper** & **RFi** & **AUC** & **ANOVA** & **Mean** & **Rank** \\ \hline _cp_ & 2 & 2 & 2 & 3 & 2 & 1 & 1 & 2 & 1 & 2 & 3 & 1 & 2 & 1.8 & 1 \\ _thal_ & 1 & 1 & 1 & 1 & 1 & 4 & 6 & 1 & 2 & 3 & 2 & 5 & 7 & 2.7 & 2  
the RWeka library [41]. In the final list of acronyms, we distinguish the implementations of each of the algorithms; if not specified, they correspond to the caret library. Regardless of the measure of quality, accuracy, or AUC metrics, in general, better results were obtained when using more features, although there were exceptions. For example, for discriminant analysis techniques (LDA--linear discriminant analysis--and FDA--Factorial Discriminant Analysis) or SVM, better results were obtained with fewer features than using all of them; this may be due to the characteristics of the classification algorithms. They tried to partition the classification space using hyperplanes to make the decision. With more features, without a mechanism that can filter or weigh them, it introduced noise when performing the partitioning of the classification space. There are algorithms that can perform progressive refinements of the model until the limit where overfitting occurs. Examples of this type are those that use boosting or bagging (Random Forest is characteristic of this behavior), where using more features produces better results because the model can be adjusted in a more complex way: the decision space is subdivided into hyper-rectangles. The disadvantage of these techniques is that they produce better results at the cost of increased difficulty of interpreting the model that guides its decisions. The decision trees are in an intermediate position: they do not produce the performance of the boosting techniques but they improve the classifiers based on hyperplane partitions, maintaining a high understanding of their models. This is why they are used in the following discussion of the behavior of complexity versus the quality of the solution. To visualize the effect of the number of features on the complexity of the results and their associated quality, we examined how AUC varied according to the depth of the tree. The AUC results of trees obtained with RPART [37] and Ctree [38] were reported. A total of 100 executions of the algorithms were performed using hold-out (70-30%) representing the mean value and the confidence interval at 95% of the AUC over the test set in Figure 6.  
RPART is an R implementation of the classification and regression trees (CART) algorithm. The algorithm is well-known and applied in two stages.