# Optimal Feature Set Size in Random Forest Regression
## 2 Feature Set Size (_mtry_)
### Random Forest Algorithm in Regression
```
0: SearchDirection \(\leftarrow\) \"forward\" or \"backward\", and \(F\gets floor(p/3)\)
0: SearchSize \(\leftarrow ceiling(p/10)\)
0:\(\hat{e}_{f}\) : out-of-bag Mean Squared Error with \(mtry=f\)
0:
1:if\(SearchDirection\) = \"backward\"then
2: Compute \(\hat{e}_{F}\)
3:for\(x\) from \((F-1)\) to \((F-SearchSize)\) with decrement 1
4:if there is \(\hat{e}_{x}\) already then skip
5:else compute \(\hat{e}_{x}\)
6:endif
7:endfor
8:\(j\gets argmin_{j}(\hat{e}_{j}\mid j\in[F-SearchSize,\cdots,F-1])\)
9:if \(\hat{e}_{j}<\hat{e}_{F}\)then\(F\gets j\) and go to 3
10:endif
11:ifSearchDirection = \"forward\"then
12: Compute \(\hat{e}_{F}\)
13:for\(x\) from \((F+1)\) to \((F+SearchSize)\) with increment 1
14:if there is \(\hat{e}_{x}\) already then skip
15:else compute \(\hat{e}_{x}\)
16:endif
17:endfor
18:\(j\gets argmin_{j}(\hat{e}_{j}\mid j\in[F+1,\cdots,F+SearchSize])\)
19:if \(\hat{e}_{j}<\hat{e}_{F}\)then\(F\gets j\) and go to 13
20:endif
21:Return \(F\)
```

**Algorithm 2** A search algorithm for estimating the best \(mtry\) in RF regression. ## 4 Experimental Study

This section studies the impact of our proposed search algorithm by comparing the RF prediction accuracy using an estimated \(mtry\) by our proposed algorithm, with that using the optimal \(mtry\), the default \(mtry\), and an estimated \(mtry\) by a typical grid search method. We determine the _SearchDirection_ by the results of the classification tree in Figure 4. Specifically,if a dataset falls into the node number 2, the _SearchDirection_ is assigned as \"backward\"; if a dataset falls into the node number 3, the _SearchDirection_ is assigned as \"forward\". ### Estimated Size vs. Optimal Size

The results of comparison between the estimated _mtry_ by our proposed algorithm and the optimal _mtry_ are presented with two figures. First, Figure 5 compares the relative distance between the estimated _mtry_ and the optimal _mtry_, where the relative distance between the optimal _mtry_ and the default _mtry_ is added above as a reference. The two subfigures clearly show that the estimated _mtry_ is closer to the optimal _mtry_ than the default, and, in 23 out of 56 datasets, the estimated _mtry_ is exactly matched to the optimal _mtry_. It demonstrates that the proposed search algorithm performs well to estimate the optimal _mtry_. Second, Figure 6 compares the relative RMSE between the estimated _mtry_ and the optimal _mtry_. Since almost all boxes are located near 0, it seems that the estimated _mtry_ and the optimal _mtry_ produce similar RF prediction performances. Figure 5: (**Top**) Comparison between the optimal _mtry_ and the default _mtry_; and (**Bottom**) comparison between the estimated _mtry_ by the proposed search algorithm and the optimal _mtry_. The _x_-axis and _y_-axis indicate 56 datasets and the relative distance, respectively. Figure 6: Comparison of the RF prediction performances using the estimated _mtry_ by the proposed search algorithm and the optimal _mtry_. The _x_-axis and _y_-axis indicate 56 datasets and the relative RMSE, respectively. Each box-plot is based on 100 relative RMSEs. ### Estimated Size vs. Default Size

Figure 7 compares the relative RMSE between the estimated _mtry_ and the default. The results show that using the estimated _mtry_ offers improvement over the default size in terms of the RF prediction because many boxes are located above 0. Interestingly, the distribution of the boxes is quite similar with that in Figure 3. It confirms that the optimal _mtry_ and the estimated _mtry_ have similar performance. We further conducted the paired t-tests based on the 100 RMSE pairs of the estimated _mtry_ and the default _mtry_ for each dataset. We observed that the RF prediction using the estimated _mtry_ is statistically better than that using the default _mtry_ on 38 out of 56 datasets, and, for the other 18 datasets, their differences are not statistically significant. There is no dataset where the default _mtry_ produces better prediction than the estimated _mtry_.