# On the Optimal Size of Candidate Feature Set in Random forest
## 4 Experimental Study
### Comparison between the Estimated and the Default Values  
According to Figure 2, there are 9 datasets in which the differences between the optimal \(m\) and the default \(m\) are larger than 3. Does the estimated value of \(m\) improve the accuracy of Random forest on these 9 outlier datasets? In Figure 6, we compared the performances between the two values. As shown in the left panel, the accuracy of the default \(m\) is statistically worse in 7 datasets among 9, compared to the optimal \(m\). Contrarily in the right panel, the accuracy of the estimated \(m\) is statistically worse in only 2 datasets. In summary, estimating the size of \(m\) nicely can provide significant improvements in more outlying cases (e.g., \(|\) optimal \(m\) - default \(m\)\(|>3\)).  
Now, we compare the accuracy of the estimated \(m\) and the default \(m\) directly. Table 3 shows the summary of comparison between the estimated \(m\) and the default \(m\) in terms of test accuracy. The values on the table represent the number of times that the vertical method was more accurate than the horizontal method. Therefore, the value 32 in the first row and second column means that using the estimated \(m\) was more accurate than using default \(m\) in 32 datasets among 58. On the other hand, there were 14 datasets where using default \(m\) performed better than using estimated \(m\). In remaining 12 datasets, the results were the same. In addition, the values inside the parenthesis represent the frequency that these differences are statistically significant from the paired t-test. The estimated \(m\) from our proposed algorithm significantly performed better than using the default \(m\) in 21 datasets. There were only 5 datasets on the opposite side.  
Figure 7 shows the confidence interval plots of the test accuracy differences between two methods. Twenty one confidence intervals were located above 0. There were only 5 intervals which were located below 0.  
Figure 5: Confidence intervals of accuracy differences between the optimal \(m\) and the estimated \(m\). The difference is significant when the confidence interval does not contain zero.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline  & & \multicolumn{2}{c}{**Win**} \\ \hline \multirow{2}{*}{**Loss**} & \multicolumn{2}{c}{Default size} & Estimated size \\  & Default size & & 32 (21) \\  & Estimated size & 14 (5) & \\ \hline \hline \end{tabular}
\end{table}
Table 3: Summary of test accuracy comparison.  
In Table 4, we conducted the randomized complete block design (RCBD) to test the overall significance of test accuracies between the estimated \(m\) and the default \(m\). The block indicates 58 datasets in which 100 replications exist. In Table 4, the \(p\)-value of the main effect was <0.0001. Therefore, we can conclude there is strong evidence that the estimated \(m\) of the proposed method is significantly better than the default \(m\) in terms of test accuracy.  
Figure 6: Confidence intervals of accuracy differences between the optimal \(m\) and the default \(m\) (**left**); between the optimal \(m\) and the estimated \(m\) (**right**).  
Figure 7: Confidence intervals of accuracy differences using the estimated \(m\) and the default \(m\). The difference is significant when the confidence interval does not contain zero.