## 2 Materials and Methods
### Application of the Ranking Algorithms on In-Silico and Real Data
#### 2.4.2 Application of the Ranking Algorithms to Real Data  
The data selected from the ELSA dataset, as detailed in Section 2.3.2, were randomly split into a training set, containing 80% of selected subjects, and a test set, containing the remaining 20% of subjects, stratifying by diabetes incidence. The training set was used for model development, while the test set was used for final model assessment. Training set variables were scaled in a range 0-1, subtracting to each variable its minimum value and then dividing it for the difference between its maximum and its minimum value. The same scaling was applied to test set data, using the minimum and maximum values of the features in the training set. Categorical variables with more than two levels (i.e., mar_stat and smoke) were re-coded using dummy variables. Physical activity (phys_act) was considered as a numerical variable.  
A Cox model for the prediction of T2D onset was trained on the training set by applying the method based on RFE and Borda count to perform variable ranking and feature selection. Before performing the ranking, we analysed the correlation between features in the training set. This analysis revealed the presence of a pair of highly correlated features: BMI and waist circumference have a Spearman correlation coefficient of 0.78. All the other pairs of features present a Spearman correlation coefficient below 0.40. As the presence of this pair of highly correlated features can affect the variable ranking, we performed the ranking considering the two ranking approaches tested in simulation:  
* Approach 1: The correlation between BMI and waist circumference is ignored and the standard RFE-Borda count method is applied;
* Approach 2: The ranking is performed with the proposed algorithm that takes into account the correlation between BMI and waist circumference.  
To better understand the impact of highly correlated features on the ranking performance of the standard RFE-Borda count method, we also tested two additional approaches in which one correlated feature is removed a priori.  
* Approach 3: Waist circumference is dropped from the analysis and the ranking is performed with the standard RFE-Borda count method, considering only BMI in the set of candidate predictors.
* Approach 4: BMI is dropped from the analysis and the ranking is performed with the standard RFE-Borda count method, considering only waist circumference in the set of candidate predictors.  
For all the approaches, the training set was resampled with replacement B = 100 times. At each iteration, the resampled training set was used to train the models with decreasing number of features as per the RFE algorithm, while the subjects of the out-of-bag set (i.e., those of the original training set not present in the resampled training set) were used to assess the performance of the models. The performance was assessed by the concordance index (C-index) [34], which measures the concordance between risk scores and event times (0.5 no concordance at all, 1 perfect concordance). The output of the RFE-Borda count method at the end of the B iterations includes:  
* a table with the mean and the standard deviation (SD) of the rank obtained for each feature across the B iterations;
* a table with the value of the C-index (mean and SD) for models with different number of features.  
Afterwards, the final feature selection is made by choosing the optimal number of features, nopt, as the number of features that maximizes the mean of the C-index across the B iterations, and then choosing from the rank table the nopt variables with highest mean rank. The final step is to train the model with the selected features on the entire training set, and to assess its performance in terms of C-index on the test set. To get a confidence level for the C-index, we also performed a 5-fold cross-validation on the training set and assessed the mean (SD) of C-index on the 5 testing folds.