# Optimal Feature Set Size in Random Forest Regression
## 2 Feature Set Size (_mtry_)
### Random Forest Algorithm in Regression
In the context of classification, [PERSON] et al. [14] and [PERSON] et al. [15] observed that the optimal size tends to be related to the number of relevant features, where the relevance means that a feature is significantly associated with the target. If there are many relevant features in a given dataset, a smaller _mtry_ value is preferred to utilize the relevant features equally for splitting nodes in the tree construction. As a result, the trees will be more diverse. In this situation, larger _mtry_ value would cause similar trees in the ensemble because the most dominant feature is repeatedly selected for the split rule at each node. On the other hand, if there are only a few relevant features in a given dataset, a larger _mtry_ value may be advantageous to increase the accuracy of individual trees. However, these observations are limited on classification problems and may not be valid in regression problems. Hence, we sought a relationship between the optimal _mtry_ and the characteristics of given datasets, focusing on the relevance of features, in the context of regression. How do we measure the relevance between the response variable \(Y\) and a feature \(X_{i}\), where \(i=1,\cdots,p\), in regression? The typical Pearson correlation coefficient between \(Y\) and \(X_{i}\) can be ineffective because it deals with linear associations only. Hence, we consider a modified Pearson correlation coefficient between \(Y\) and \(\hat{Y}_{i}\), which is a predicted response for \(X_{i}\), obtained by a regression tree between \(Y\) and \(X_{i}\). In detail, a decision tree between \(Y\) and \(X_{i}\) is fitted with pruning, and then the predicted response \(\hat{Y}_{i}\) is obtained by applying \(X_{i}\) on the fitted tree. We can capture both linear and nonlinear associations by considering the decision tree [2]. We denote the modified Pearson correlation coefficients as \(R_{i}\), \(i=1,\cdots,p\), and the larger \(R_{i}\) means stronger evidence that \(X_{i}\) is a relevant feature to the response \(Y\). To investigate whether the optimal _mtry_ is related to the relevance of features, we consider a classification tree, where the target variable \(Y_{dt}\) is a binary outcome that is \(1\) if the optimal size is larger than the default size, and is \(0\) if the optimal size is smaller than or equal to the default size. We also create a variety of factors \(X_{dt}\) to be used in the decision tree: (1) _nrel_, number of relevant features; (2) _nirr_, number of irrelevant features; (3) _scor_, standard deviation of \(R_{i}\); (4) _mcor_, mean of \(R_{i}\); (5) _ccor_, coefficient of variation of \(R_{i}\); and (6)

Figure 3: Comparison of the RF prediction accuracies using by the optimal _mtry_ and the default _mtry_. The \(x\)-axis indicates 56 datasets with the same order as in Figure 2. The \(y\)-axis indicates the relative RMSE. Each box-plot is based on 100 relative RMSEs. \(pn\), ratio of features and observations. The definitions of \(Y_{dt}\) and \(X_{dt}\) are summarized in Table 3.

Figure 4 depicts the classification tree with pruning between \(Y_{dt}\) and \(X_{dt}\) based on a total of 56 datasets, where \(scor\), standard deviation of \(R_{i}\), is the most significant factor to discriminate the 56 datasets. The result shows that, for \(scor<0.135\), there are many datasets with the optimal size less than or equal to the default size; for \(scor\geq 0.135\), most datasets have an optimal size larger than the default size. These results can be interpreted as follows. If features in a given dataset have similar relevance to each other, the optimal size tends to be small, thus setting smaller \(mtry\) value can be helpful to achieve good prediction performance. Conversely, if features are widely distributed from irrelevant to relevant, the optimal size tends to be large, thus setting larger \(mtry\) value may be preferred. We found that the optimal feature set size may be related to the relevance of features in a given dataset, but it is not enough to select a specific \(mtry\) in RF model fitting.