# Variable Selection in Time Series Forecasting Using Random Forests
## 2 Methods and Data
### Methods
#### Random Forests  
RF is a vague term [20]. Here we use the original RF algorithm introduced in [18]. In particular, we present the version of the algorithm used for regression. The specific version of the algorithm has been implemented in the randomForest R package [60]. The remainder of the section is largely reproduced from [20] with adaptations, while identical notation is used in [20,60]. The present Section is not intended to provide a detailed description of RF. The interested reader can find further details in [20], while the computations are reproducible (Appendix A). We assume that the \(\underline{u}\) is a random vector with \(k\) elements. The aim is to predict \(v\) by estimating the regression function:  
\[m(\underline{u})=\text{E}[\underline{v}\mid\underline{u}=\underline{u}] \tag{18}\]given the fitting sample:  
\[S_{s}=\left((\mathbf{u}_{1},\,v_{1}),\,\ldots,\,(\mathbf{u}_{s},\,v_{s})\right) \tag{19}\]  
which are independent realizations of the random variable \((\mathbf{\underline{u}},\,\underline{v})\). Therefore, the aim is to construct an estimate \(m_{s}\) of the function \(m\). A random forest is a predictor constructed by growing \(M\) randomized regression trees. For the \(j\)-th tree in the family, the predicted value at \(\mathbf{u}\) is denoted by \(m_{s}(\mathbf{u};\,\underline{\theta}_{j},\,S_{s})\), where \(\underline{\theta}_{1},\,...,\,\underline{\theta}_{M}\) are independent random variables, distributed as \(\underline{\theta}\) and independent of \(S_{s}\). The random variable \(\underline{\theta}\) is used to resample the fitting set prior to the growing of individual trees and to select the successive directions for splitting. The prediction is then given by the average of the predicted values of all trees. Before constructing each tree, \(b_{s}\) observations are randomly chosen from the elements of \(\mathbf{u}\). These observations are used for growing the tree. At each cell of the tree, a split is performed by maximization of the CART-criterion (defined in [20]) by selecting mtry variables randomly among the \(k\) original ones, picking the best variable/split point among the mtry and splitting the node into two daughter nodes. The growing of the tree is stopped when each cell contains fewer than nodesize points. The parameters used in RF are \(b_{s}\in\{1,\,\ldots,\,s\}\), mtry \(\in\{1,\,\ldots,\,k\}\), nodesize \(\in\{1,\,\ldots,\,b_{s}\}\), and \(M\in\{1,\,2,\,\ldots\,\,1\}\). In most studies, it is agreed that increasing the number of trees does not decrease the predictive performance; however, it results in an increase of the computational cost. [PERSON] et al. [32] suggests a range between 64 and 128 trees in a forest based on experiments. [PERSON] and [PERSON] [34] (p. 200) suggest using at least 1000 trees. [PERSON] and [PERSON] [33] suggest that the highest performance gain is achieved when training 100 trees. In the present study, we use \(M=500\) trees. The number of sampled data points in each tree and the number of examples in each cell below which the cell is not split are set equal to their default values in the randomForest R package, i.e., equal to the sample size and nodesize = 5, respectively. These are reported to be good values [20; 35]. The parameter mtry is estimated during the validation phase by implementing the trainControl function of the R package caret [50; 51]. The optimal mtry is found by using bootstrap resampling, i.e., randomly sampling the validation test with reselection (the size of the bootstrap sample must be equal to the size of the validation test), fitting the model in the bootstrap sample and measuring its performance in the remaining set, i.e., the data not selected by the bootstrap sampling. Twenty-five iterations are used in the resampling procedure. The optimal mtry is found by calculating the average performance of the fitted models, while the search for the optimal value of the parameter is performed in a grid. The root mean squared error (RMSE) is used to measure the performance. The optimal mtry minimizes the RMSE.