## 2 Methodology
### Feature Selection Methods Based on Machine Learning
#### 2.1.2 Feature Selection with Random Forest  
Random forest (RF) is an ensemble machine learning method using tree-type classifiers. It is built by bootstrap sampling technology and random splitting technology, and the final classification result is made by a majority vote of the trees [39; 40]. Because of its excellent generalization performance, RF is also further used for feature selection [41; 42]. For a given tree, let \(S_{0}\) denote the set of input predictor data vectors and \(S_{j}\) be the subset of the predict data reaching node \(j\) in the binary split tree. According to the performance of the current feature on node \(j\), \(S_{j}\) can be divided into two subsets, i.e., \(S_{j}^{L}\) and \(S_{j}^{R}\); here, \(S_{j}^{L}\bigcup S_{j}^{R}=S_{j}\) and \(S_{j}^{L}\bigcap S_{j}^{R}=\emptyset\). Choosing the best split according to the mean decrease of the Gini index, which is defined as:\[\Delta Gini_{i}(j)=Gini(j)-(\frac{|S_{j}^{L}|}{|S_{j}|}Gini(j^{L})+\frac{|S_{j}^{R} |}{|S_{j}|}Gini(j^{R})) \tag{2}\]  
where \(Gini(j)=1-\sum_{c\in C}P_{c}^{2}\) is the Gini index at node \(j\). This metric reflects the contribution of each feature to node \(j\); therefore, we can get an estimate of feature \(i\) with Gini importance:  
\[Score_{Gini}(i)=\frac{1}{n_{tree}}\sum_{t=1}^{n_{tree}}\sum_{j}\Delta Gini_{i}( j,t) \tag{3}\]  
where \(\Delta Gini_{i}(j,t)\) is the value of \(\Delta Gini_{i}(j)\) on one tree \(t\). The Gini importance indicates how large its overall discriminative value is for the classification task. We randomly chose a feature \(i\), calculated its Gini importance defined in (4) and removed the features with Gini importance below feature \(i\). The algorithm for feature section with random forest by Gini importance (RFFS-GI) is briefly described as below. ```
0:
1. Randomly choose a feature \(i\) into the feature set;
2. Calculate the Gini importance of all features in the feature set with the scoring function (3);
3. Keep features with Gini importance above that of the feature \(i\);
0: Optimal feature subset
```  
**Algorithm 2**Feature section with random forest by Gini importance (RFFS-GI)  
In addition, for bootstrap sampling technology, about \(1/3\) of the samples will not be collected at the end, and they are called the out of bag (OOB) data [43]. The role of OOB data can be considered as equivalent to the test data. Therefore, we can also use the classification accuracy of the random forest classifier on the OOB data as the feature separability criterion, so as to calculate the importance of each feature:  
\[Score_{OOB}(i)=\sum\frac{ooberr2-ooberr1}{N} \tag{4}\]  
where \(ooberr1\) is the classification error of the OOB data, \(ooberr2\) is the classification error of the OOB data with adding noise on feature \(i\) and \(N\) indicates the number of trees in a random forest. We can understand that if a feature is randomly disturbed, the classification error of the OOB data will increase greatly, and it can be considered that this feature has a great influence on the classification result. The algorithm of feature section with random forest by the classification accuracy on the OOB data (RFFS-OOB) is briefly described as below. ```
0:
1. Generate random forest;
2. Calculate feature importance based the scoring function (4), and sort the scores;
3. The top ranked features are selected as the optimal feature subset. 0: Optimal feature subset. ```  
**Algorithm 3**Feature section with random forest by the classification accuracy on the OOB data (RFFS-OOB)  
In addition, for bootstrap sampling technology, about \(1/3\) of the samples will not be collected at the end, and they are called the out of bag (OOB) data [43]. The role of OOB data can be considered as equivalent to the test data.