# Hybrid-Recursive Feature Elimination for Efficient Feature Selection
## 1 Introduction  
As datasets continue to increase in size, it is important to select the optimal subset of features from a raw dataset in order to obtain the best possible performance in a given machine learning task. An efficient and small feature (variable) subset is especially important for building a classification model. High-dimensionality datasets can easily cause overfitting problems, in which case a reliable model cannot be obtained. Furthermore, such datasets require high computing power and large volumes of storage space [1], and often produce models with low classification accuracy. This is called the \"curse of dimensionality\" [2]. Thus, it is necessary to select a representative subset of features to solve these problems.  
Feature selection [3; 4; 5; 6] has become necessary to select the best subset of features, and is used in various fields including biology, medicine, finance, manufacturing and production, and image processing. Recursive feature elimination (RFE) is a feature selection method that attempts to select the optimal feature subset based on the learned model and classification accuracy. Traditional RFE sequentially removes the worst feature that causes a drop in \"classification accuracy\" after building a classification model. A new RFE approach was recently proposed which evaluates \"feature (variable) importance\" instead of \"classification accuracy\" based on a support vector machine (SVM) model, and chooses the least important features for elimination [7]. This approach can be applied to other classification models such as random forests (RFs) and gradient boosting machines (GBMs), both of which have in-built feature evaluation mechanisms.  
In Figure 1, the overall process for selecting features using the feature-importance-based RFE method is shown. When a classifier is trained with a training dataset, feature weights that reflect theimportance of each feature can be obtained. After all features are ranked according to their weights, the feature that has the lowest weight value is removed. Then, the classifier is re-trained with the remaining features until it has no features left with which to train [7]. Finally, the entire ranking of the features using the feature-importance-based RFE method can be obtained. This approach is an embedded feature selection method [7] and has been shown to provide strong performance, compensating for the weaknesses of the filter [8] and wrapper methods [9].  
[PERSON] [7] proposed an **SVM-based RFE (SVM-RFE)** algorithm to evaluate the importance of each feature. The SVM aims to find the hyperplane that divides classes the most. Kernel techniques in the SVM ensure that high-dimensional data are well separated. The values of the feature weight vector are obtained using the linear kernel, and the weight values are used to evaluate the importance of the features. Suppose \(D(\mathbf{x})\) is the decision function for the hyperplane, and \(c\) represents the number of classes. If the data has multiple classes (i.e., more than two), \(q\), which denotes the entire number of hyper-planes, is calculated according to the equation \(q=c(c-1)/2\). Equation (1) refers to the decision function for the binary class case, and Equation (2) is for a multi-class dataset.  
\[D(\mathbf{x})=sign(\mathbf{x}\star\mathbf{w}) \tag{1}\]  
Figure 1: Process of feature-importance-based recursive feature elimination (RFE). N denotes the number of features.