# Optimal Feature Set Size in Random Forest Regression
## 2 Feature Set Size (_mtry_)
### Random Forest Algorithm in Regression
Hence, in the next section, we develop a search algorithm for estimating the best \(mtry\) in RF regression. ## 3 Search Algorithm for Optimal Feature Set Size in Random Forest Regression

We develop a search algorithm for estimating the best \(mtry\) in RF regression using the findings in Section 2. This algorithm is also motivated by [PERSON] and [PERSON] [16]. Figure 4: Classification tree with pruning between \(Y_{dt}\) and \(X_{dt}\) based on a total of 56 datasets. At each node, the first line indicates the predicted class, the second line means the proportion of classes. The last line indicates the ratio of observations belonging to the node among the total observations. \begin{table}
\begin{tabular}{l c c} \hline
**Attribute** & **Name** & **Definition** \\ \hline Target (\(Y_{dt}\)) & \(Y_{dt}\) & \(Y_{dt}=1\) if Optimal \(mtry>\) Default \(mtry\) \\  & & \(Y_{dt}=0\) if Optimal \(mtry\leq\) Default \(mtry\) \\ \hline \multirow{5}{*}{Factor (\(X_{dt}\))} & \(nrel\) & \(Length(R>Q_{3}(R))/p\) \\  & \(nirr\) & \(Length(R<Q_{1}(R))/p\) \\  & \(scor\) & \(Sd(R)\) \\  & \(mcor\) & \(Mean(R)\) \\  & \(ccor\) & \(Sd(R)/Mean(R)\) \\  & \(pn\) & \(p/n\) \\ \hline \end{tabular}
\end{table} 
Table 3: Description of \(Y_{dt}\) and \(X_{dt}\) for classification tree analysis. The definitions of \(X_{dt}\) are as follows: (1) \(nrel\), the number of relevant features (\(Q_{3}(R)\): the third quartile of \(R\)); (2) \(nirr\), the number of irrelevant features (\(Q_{1}(R)\): the first quartile of \(R\)); (3) \(scor\), the standard deviation of \(R\); (4) \(mcor\), the mean of \(R\); (5) \(ccor\), the coefficient of variation of \(R\); and (6) \(pn\), the ratio of features and observations. Note that \(R\) is the collection of \(R_{i}\), \(i=1,...,p\). Our proposed algorithm combines a typical grid search method and two unique concepts: (1) _SearchDirection_, which controls the direction of search, \"forward\" and \"backward\"; and (2) _SearchSize_, which decides how many features are searched at a time, set \(ceiling(p/10)\) by default. These two concepts make our proposed search algorithm more efficient than the typical grid search method by reducing the number of searches. The proposed algorithm searches using the out-of-bag mean squared error (OOB-MSE), which is computed on the out-of-bag samples. The OOB-MSE estimate is known as a good estimate for the true MSE [1]. In the _SearchDirection_ = \"forward\" setting, the proposed algorithm starts the searches from \(mtry=floor(p/3)\) to \(p\) in increasing order. Note that, if _SearchDirection_ is \"backward\", the searches are performed from \(mtry=floor(p/3)\) to 1 in decreasing order. We now describe our proposed search algorithm using a simple example. Suppose that _SearchDirection_ is \"forward\" and a dataset has 15 features (\(p=15\)); then, _SearchSize_ is set as \(ceiling(15/10)=2\) by default. The proposed method first computes the OOB-MSE of the RF model with \(mtry=(15/3)=5\), which is denoted as \(\hat{e}_{5}\). Then, it compares \(\hat{e}_{5}\) with the next two OOB-MSEs, \(\hat{e}_{6}\) and \(\hat{e}_{7}\), which are obtained by the RF model with \(mtry=6\) and \(mtry=7\), respectively, given that _SearchSize_ is 2. If \(\hat{e}_{5}\) is smaller than the minimum of \(\hat{e}_{6}\) and \(\hat{e}_{7}\), the algorithm estimates the best \(mtry\) as 5 and terminates the search. If \(\hat{e}_{5}\) is larger than one of \(\hat{e}_{6}\) and \(\hat{e}_{7}\), the minimum of \(\hat{e}_{6}\) and \(\hat{e}_{7}\) is continuously compared with the next two OOB-MSE, \(\hat{e}_{8}\) and \(\hat{e}_{9}\). The algorithm is reiterated until stopping criteria are met. The detailed search algorithm is in Algorithm 2.