# Feature Importances: A Tool to Explain Radio Propagation and Reduce Model Complexity
## 3 Problem Description and Model Used
### Relative Feature Importances in Tree-Based Models  
Feature selection is a crucial element of the design of every machine learning model [23]. It can be performed based on the ranking of features, according to their contribution to the model's predictions. A number of specific methods which calculate feature importance have been developed. An exhaustive approach towards this goal would involve first training the model according to every possible subgroup of features, and then comparing the estimation results to determine the most efficient subgroup of features. Such an approach would need a vast amount of computer resources and would also be extremely time-consuming.  
Another approach, which is implemented in the current study, is one that takes advantage of the built-in ability of most tree-based models to rank features (also denoted as predictors) according to their participation in the process of node splitting during the procedure of building the regression trees.  
Each node of a regression tree corresponds to one of the predictors, combined with a cutpoint (or split point) of the aforementioned predictor. The terminal nodes hold the value of the output variable. That is, for every given set of predictors' values, a specific path through the nodes is followed, finally leading to a terminal node where the prediction is contained; that is, the tree segregates the predictor space (which is the set of all possible values for all predictors) to make its predictions. Every added node divides the predictor space further.  
Node splitting can be described as a procedure in which [22] the predictor space is divided into two distinct and non-overlapping regions in such a way that the residual sum of squares is minimized. This means that all predictors, in addition to all possible cutpoint values for each predictor, are considered to perform the split. The procedure terminates when a preselected criterion (mostly referring to the tree's maximum depth or the minimum number of training examples corresponding to a single node) is met.  
Mathematically speaking, for any predictor \(j\) and any cutpoint \(s\), the following pair of half-planes is defined:  
\[R_{1}(j,s)=\left\{X|X_{j}<s\right\}\text{\emph{and} }R_{2}(j,s)=\left\{X|X_{j}\geq s\right\} \tag{1}\]  
where the notation \(\left\{X|X_{j}<s\right\}\) denotes the region of the predictor space in which the observation \(X_{j}\) takes on a value less than \(s\).  
The values of \(j\) and \(s\) are determined such that the following quantity (residual sum of squares) is minimized:  
\[\sum_{i\
eq x_{i}\in R_{1}(j\
eq s)}\left(y_{i}-\hat{y}_{R_{1}}\right)^{2}+\sum_ {i\
eq x_{i}\in R_{2}(j\
eq s)}\left(y_{i}-\hat{y}_{R_{2}}\right)^{2} \tag{2}\]  
where \(\hat{y}_{R_{1}}\) is the mean response for the training observations \(x_{i}\) in the region \(R_{1}\)(\(j\),\(s\)) and \(\hat{y}_{R_{2}}\) is the mean response for the training observations \(x_{i}\) in the region \(R_{2}\)(\(j\),\(s\)). The actual output for the \(i\)th input pattern is denoted \(y_{i}\).  
It is therefore straightforward to claim that node splitting improves the tree's predictions. The sum of the improvements over all nodes of a tree for which the splits were made according to a particular feature is the relative importance of that feature. This measure of importance is then easily generalized to the whole ensemble of trees by calculating it for each individual tree and averaging it for the total number of trees.