# A Feature Selection Algorithm Performance Metric for Comparative Analysis
## 3 Fitness Function  
A composite fitness function is used to measure the quality of a candidate solution in this study. The fitness function takes into account the two primary objectives of feature selection, namely, the maximisation of classifier performance and minimisation of the number of features used. In this study, a solution \(s\) to the feature selection problem is encoded as a binary string of length \(N\) (the number of features in the dataset), where 1 indicates inclusion and 0 indicates exclusion of the feature. The fitness function is formulated as  
\[f(s)=f_{c}(s)k_{c}-f_{p}(s)k_{p}, \tag{1}\]  
where \(f_{c}(s)\) is the classifier performance function and \(f_{p}(s)\) is a penalty function scaled exponentially on the number of features of the solution. The variables \(k_{c}\) and \(k_{p}\) are scaling constants. The intention of the scaling constants is to determine the priority of each feature selection objective respectively. The values of these scaling constants are determined in Section 5. The sensitivity of the scaling constants is examined in Section 7. The fitness function in Equation (1) bears a resemblance to the Akaike information criterion (AIC) [16] for model selection. The AIC is formulated as  
\[AIC=2k-2\ln{(\hat{L})}, \tag{2}\]  
where \(k\) is the number of parameters to the model and \(\hat{L}\) is the likelihood function of the model. Lower AIC values are considered better where higher values for the fitness function in Equation (1) are considered better. Equation (1) and the AIC both consist of goodness of fit and penalty terms. The key difference between AIC and the fitness function in Equation (1) lies in the penalty terms. AIC uses a linear scale for the number of  
Figure 1: Illustration of the algorithm selection model component interactions [6]. features selected, i.e., the value of \(k\). Equation (1) uses an exponential scale. An exponential penalty scale is desirable for feature selection since the feature selection problem solution space has a size of \(|S|=2^{N}\), where \(N\) represents the number of features for the problem. The behaviour and intent of AIC is also different to that of the fitness function presented in Equation (1). The AIC establishes the quality of a model with respect to a collection of other candidate models: a many-to-many relationship. The fitness function in Equation (1) establishes the quality of a model with respect to the dimensionality of the problem on a per problem basis: a many-to-one relationship. The performance of the classifier, \(f_{c}(s)\), is measured using the \(F\)-measure [17]. The \(F\)-measure computes a score from the combination of the precision and recall accuracies of the model. The overall classifier performance is reported as the unweighted micro-average [18] of the \(F\)-measure. A classifier has its own bias with regard to features that are considered useful to the classifier [19; 20]. As a result of classifier bias, wrapper techniques will select different features for different classifiers on the same dataset [19]. A recent study by [PERSON] et al. [20] investigated the importance of the classifier for feature selection, concluding that the choice of classifier directly affects the computation cost and algorithm performance. The classic \(k\)-NN [21] using Euclidean distance, a simple non-stochastic classifier, is used as the underlying classifier to calculate \(f_{c}(s)\). For this reason, although the performance results in the experiment of Section 7 cannot be generalised to other classifiers, the main result confirming the presence of performance complementarity should hold for other classifiers as well. The classifier implementation makes use of the Weka [22] library. \(k\)-NN is chosen as a classifier since it is a very simple algorithm that does not require excessive training. The best value for \(k\) is problem dependant. The focus of this study is on the evaluation of the feature selection algorithms, not of the underlying classifer. Therefore, \(k\) is arbitrarily chosen as \(k=3\).