# A Model-Agnostic Algorithm for Bayes Error Determination in Binary Classification
## 5 Application of the ILD Algorithm to the Framingham Heart Study Dataset  
In this section, a practical application of the ILD algorithm is illustrated. The application to a real dataset has the purpose of giving an intuitive understanding of the method and highlight its power. Here, the medical dataset named Framingham [23] were chosen, which is publicly available on the Kaggle website [25]. This dataset comes from an ongoing cardiovascular risk study made on residents of the town of Framingham (Framingham, MA, USA). Different cardiovascular risk score versions were developed during the years [26], the most current of whom is the 2008 study in [27], to which the ILD algorithm results are also referred for comparison of performances. The reader interested in descriptive statistics on the dataset's features (averages, standard deviations, counts, description, etc.) can find all information in [23]. The classification goal is to predict, given a series of risk factors, the 10-years risk of a patient of future coronary heart disease. This is a high impact task, as 17.9 million deaths occur worldwide every year due to heart diseases [28] and their early prognosis may be of crucial importance for a correct and successful treatment. Note that the dataset available in [25] has 16 features. To make the comparison as meaningful as possible, seven of the eight features used in the original study [23] were selected. The high-density lipoprotein (HDL) cholesterol variable is unfortunately missing with respect to the original Framingham dataset employed in [27] and therefore could not be used. Thus, the dataset used in this study contains 4238 patients and 7 features: gender (0: female, 1: male); smoker (0: no, 1: yes); diabetes (0: no, 1: yes); hypertension treatment (0: no, 1: yes); age; total cholesterol; and systolic blood pressure (SBP). The last three features are continuous variables and were discretized as follows:  
* age: 0 if age \(<\) 40 years, 1 if age \(\geq\) 40 years and age \(<\) 60 years, 2 if age \(\geq\) 60 years;
* total cholesterol: 0 if total cholesterol \(<\) 200 mg/dL, 1 if total cholesterol \(\geq\) 200 mg/dL and total cholesterol \(<\) 240 mg/dL, 2 if total cholesterol \(\geq\) 240 mg/dL;
* SBP: 0 if SBP \(<\) 120 mmHg, 1 if SBP \(\geq\) 120 mmHg. The outcome variable is binary (0: no coronary disease, 1: coronary disease). Finally, to create the buckets all missing values are substituted by a feature value of \(-1\). The application of the algorithm starts with the population of the buckets as described in the previous sections. A total number of 177 buckets are populated. The dataset is split into two parts: a training set \(S_{T}\) (80% of the data) and a validation one \(S_{V}\) (20% of the data). For comparison, the Naive Bayes classifier is also trained and validated respectively on \(S_{T}\) and \(S_{V}\). The performance of the ILD algorithm and the Naive Bayes classifier are shown through the ROC curve in Figure 5. The AUC for the ILD algorithm is 0.78, clearly higher than that for the Naive Bayes classifier, namely, 0.68. To further test the performance, the split and training is repeated for 100 different dataset splits. Each time both the ILD algorithm and the Naive Bayes classifier are applied to the validation set \(S_{V}\) and the resulting AUCs are plotted in Figure 6 (top panel). For clarity, the difference between the AUC provided by the two algorithms is shown in Figure 6 (bottom panel). This example shows that the application of the ILD algorithm allows the comparison of the prediction performance of a model, here the Naive Bayes classifier, with the maximum obtainable for a given dataset. The maximum accuracy over the validation set \(S_{V}\) is 85% for the Naive Bayes classifier (calculated for a specific threshold, i.e., 61%, which optimizes the accuracy over the training set \(S_{T}\)) and 86% for the ILD algorithm (calculated applying Theorem 1).