## 4 Methods
### Features Selection and Classification  
An important step during a radiomics approach is the features selection step: it helps in reducing the dimensionality of the problem, lowering the curse of dimensionality and the risk of overfitting, and it helps finding the most informative set of descriptors for the problem at hand. The feature selection pipeline, represented in the blue, purple and red blocks of Figure 2, consists in a wrapper-based approach, which searches and evaluates the best subset of features maximising the performance of a given classification algorithm. To avoid any bias, features are normalised before each step, using a standard scaler, as represented in Figure 2. This approach scales features while using the following equation: \(z=(x-\mu)/\sigma\), with \(\mu\) mean value, \(\sigma\) standard deviation and with \(x\) and \(z\) as the original feature and the scaled feature, respectively. Starting from the blue box in Figure 2, the feature set was analysed by the Recursive Feature Elimination (RFE) approach [51], making use of four different classification algorithms, namely AdaBoost, CART Decision Tree (DT), Random Forest (RF), and XGBoost (XGB) [52]. The use of four learners and the availability of three different types of VOIs gave rise to 12 runs of the wrapper, which are listed in Table 2.  
The goal of the RFE is to select a feature subgroup, by initially considering all of the features and recursively examining a smaller and smaller dataset according to an assigned feature importance attribute. During this process, the selected classifier is trained on the standardised descriptor set to compute the feature importance attribute based on the accuracy level of the model. The feature with the lowest attribute is excluded from further analysis. This recursive analysis is performed until the best feature is found resulting in a final descriptor ranking. To avoid any bias during the features selection approach, the RFE step was applied according to a nested leave-one-out (LOO) procedure: indeed, referring to Figure 2, there is an outer grey loop for final performance computation and an inner blue loop for RFE application. This means that this analysis was repeated \(N-1\) times, to obtain a global view of the feature importance. Hence, all of the \(N-1\) rankings obtained were summed up to get a final rank of each descriptor. From this step on we considered two alternative paths referred to as single ranking and total ranking. On the one hand, in the first path, the ranking procedure was performed independently for each single experiment, i.e., considering a single classifier and a single type of VOI. Straightforwardly, each of the 12 experiments listed in Table 2 uses a specific features rank and, in turn, this permits us to optimise the feature set for each learning algorithm and for each VOI. On the other hand, in the second path, the wrapper first computes the rank of the features for the GTV, CTV, and PTV; then, such rankings computed using the same classifier on the three volumes  
Figure 3: Visual representation of a TOP-LBP example: each ellipsoid shows a planar LBP calculated on a specific orthogonal plane. were summed up to gain a global ranking for every learning algorithm. This implies that we finally have four different feature sets, one for each classifier used in the wrapper, regardless of the VOI used. This permitted designing experiments that are comparable within the same classifier. Turning our attention to the purple and red boxes in Figure 2, these represent the feature evaluation workflow to find the best subset and it was performed independently for both single ranking and total ranking analyses. To find out the best descriptors, we tested different combinations of features subsets whose performance was evaluated according to a given metric. To this goal, all of the features were initially sorted by their rank and, then, the subsets were sequentially inspected using a cumulative forward selection approach (yellow loop).