# Research on Ensemble Learning-Based Feature Selection Method for Time-Series Prediction
## 3 Feature Selection Based on Ensemble Learning
### Determination of Ensemble Learning Strategies  
Given the set of feature selection methods \(\{M_{1},M_{2},\ldots,M_{n}\}\), an appropriate ensemble learning strategy is needed to obtain the final postfiltered feature subset \(S_{\text{final}}\). To integrate information from multiple feature selection methods, mitigate dependence on a single method, and enhance overall robustness, this study adopts the feature scoring weighting approach as its ensemble learning strategy. The feature scoring weighting approach exhibits comprehensive, flexible, and performance-enhancing characteristics across various problems and data contexts. This method facilitates the amalgamation of information from multiple feature selection methods, thereby improving model performance and robustness, diminishing the risk of overfitting, all while upholding a certain level of interpretability. The specific implementation steps are outlined as follows:  
Firstly, through each feature selection method \(M_{i}\), the score \(S_{i,j}=f_{i,j}(X,Y)\) for each feature is calculated according to its respective feature scoring strategy. Here, \(s_{i,j}\) represents the score assigned by method \(M_{i}\) to feature \(j\).  
To ensure comparability among importance scores under different feature selection methods, it is imperative to constrain the scores of features within the range of 0 to 1. Therefore, normalization of the feature scores provided by each method is requisite. This paper employs a softmax transformation for the normalization of importance scores.  
In this way, the raw feature score \(s_{i}\) is transformed into a probability representing each feature, ensuring that they range between 0 and 1, with a total sum of 1. These normalized scores can be utilized as comparable values across different feature selection methods.  
Different feature selection methods may have varying applicability in different domains and for different types of time-series data. In order to comprehensively consider the scores from multiple methods, this paper introduces a weight vector \(\text{W}=[w_{1},w_{2},\ldots,w_{n}]\)  
Figure 1: Flowchart of feature selection based on ensemble learning.  
where \(w_{i}\) represents the weight assigned to method \(M_{i}\). These weights are utilized to adjust the relative contributions of each method.  
This paper employs the K-fold cross-validation method to determine the weights for each feature selection method. This method divides the dataset into K folds and performs the following steps for each fold: First, select K \(-\) 1 folds as the training set, and reserve one fold as the test set. Then, apply each feature selection method \(M_{i}\) on the training set, obtaining the feature selection results \(S_{i}\) for each method. Next, train the regression model using the filtered feature subset and evaluate the model's performance on the test set. Mean Squared Error (MSE) is used as the performance metric for each method and is recorded. For each feature selection method \(M_{i}\), based on the performance metric results from K-fold cross-validation, calculate the average performance metric \(avg_{i}\). These average performance metrics serve as indicators for the weights \(w_{i}\). The weights \(w_{i}\) are then normalized based on the average performance metrics to ensure that they sum up to 1, i.e.,  
\[w_{i}=\frac{avg_{i}}{\sum_{i=1}^{5}avg_{i}} \tag{3}\]  
Finally, the learned weights \(w_{i}\) are utilized in the output of the feature scoring weighting method, forming the ultimate feature selection results:  
\[S_{\text{final}}\ =\sum_{i=1}^{n}w_{i}\cdot S_{i} \tag{4}\]  
where \(S_{\text{final}}\) represents the ultimate feature subset, which is the weighted sum of scores from each method. The weight vector W governs the relative importance of each method. The final feature subset \(S_{\text{final}}\) incorporates information from multiple method scores, representing the features ultimately selected.