# Optimal Feature Set Size in Random Forest Regression
## 2 Feature Set Size (_mtry_)  
### Random Forest Algorithm in Regression  
RF, a popular ensemble machine learning method, is a modified version of bagging [20] since they share conceptually same algorithms to each other, but the main difference is that RF further reduces the variance of a prediction model by combining more de-correlated trees than bagging. The RF algorithm starts with generating bootstrapped samples from a training dataset. Multiple regression trees are fitted on the bootstrapped samples. When the trees are constructed, RF uses a random feature subset instead of all features to find the best split rule at each split of trees. The results of each tree are combined to produce a final prediction. The detailed RF algorithm in regression is described in Algorithm 1. ```
Training Phase : Given : - \(D\) : training set with \(n\) observations, \(p\) features, and the response variable. - \(B\) : number of regressors in the ensemble. Procedure :  For \(b\) = 1 to \(B\)  1. Generate bootstrapped sample \(D_{b}^{*}\) from training set \(D\). 2. Grow a regression tree using the bootstrapped sample \(D_{b}^{*}\). For a given node \(t\),  (i) Randomly sample \(mtry\) features from the full features. (ii) Find the best split rule using the random feature subset. (iii) Split the node \(t\) into two child nodes using the best split rule. Repeat (i)-(iii) until stopping rules are met. 3. Obtain a trained regressor \(R_{b}\). Test Phase :  For a test instance \(x\), the prediction estimated by the \(B\) regressors is given as : \[R(x)=\frac{1}{B}\sum_{b=1}^{B}R_{b}(x)\]

**Algorithm 1** The RF algorithm in regression. ### Influence on Prediction Performance

In the literature, three \(mtry\) values, \(1\), \(log_{2}(p)+1\), and \(p/3\), have been primarily used in the RF model fitting. The first two were introduced by [PERSON] [1], and the last was recommended for regression problems by [PERSON] et al. [2] and [PERSON] [21]. We consider \(p/3\) as the default \(mtry\) value throughout this paper. We conducted experimental studies to investigate the influence of the feature set size on the RF prediction accuracy in regression. The experiments were based on 56 real or artificial datasets that were used in other studies or came from the UCI data repository [22]. Tables 1 and 2 contain information about the datasets. \begin{table}
\begin{tabular}{l c c c} \hline \hline
**Name** & **Observation** & **Feature** & **Source** \\ \hline aba & 4177 & 8 & UCI (Abalone) \\ air & 1503 & 5 & UCI (Airfoil Self-Noise) \\ ais & 202 & 12 & [23] \\ alc & 2462 & 18 & [24] \\ ame & 3044 & 21 & [25] \\ app & 19,735 & 25 & UCI (Appliances energy prediction) \\ att & 838 & 9 & [26] \\ aut & 398 & 7 & UCI (Auto MPG) \\ beh & 135 & 17 & UCI (Behavior of the urban traffic in Sao Paulo) \\ blo & 208 & 134 & R Library _caret_ (Blood-Brain Barrier) \\ bos & 506 & 13 & R Library _mlbench_ (_Boston Housing data_) \\ bre & 198 & 32 & UCI (Breast Cancer Wisconsin) \\ bsd & 731 & 11 & UCI (Bike Sharing with count daily) \\ bsh & 17,379 & 12 & UCI (Bike Sharing with count hourly) \\ bud & 1729 & 10 & [27] \\ can & 3775 & 9 & [28] \\ car & 804 & 17 & R Library _caret_ (cars) \\ cdo & 378 & 9 & [29] \\ com & 1994 & 100 & UCI (Communities and Crime) \\ con & 1030 & 8 & UCI (Concrete Compressive Strength) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The description of 56 datasets for regression problems. The design of the experiments is as follows: we randomly split a dataset into 60% training set for fitting and 40% test set for evaluating. For the RF model fitting, we used the full range of values from 1 to \(p\) as the value of _mtry_. We set the number of trees as 100, which is a common choice in many RF applications [15,16,43]. For evaluation, root mean squared error (RMSE) was calculated to measure the RF prediction performance. All experiments were repeated 100 times to obtain stable results. The results are organized with three figures. First, Figure 1 compares the RMSE depending on different _mtry_ values using four representative datasets.