# A Feature Selection Algorithm Performance Metric for Comparative Analysis
## 5 Fitness and BFI Robustness  
An artificial dataset was constructed to test the robustness of the fitness function and BFI measure. The artificial dataset consists of 20 data instances, with four features each. The features and classes that are used in the artificial dataset are described in Table 2.  
F1 is a unique ID that does not have any information to correlate with the class. F2 and F3 are completely correlated with the class, but offer completely redundant information. F4 is a sequence of zeros, exhibiting no information related to the class. Complete correlation means that the feature has discriminatory information with respect to the class, which will always result in correct classification if selected. Table 3 reports a truncated example of the artificial dataset. The fitness function scaling constants, \(k_{p}\) and \(k_{c}\), need to be determined in line with the importance of the feature selection objectives. In order to determine the correct scaling constants for the fitness function, a set of test cases was created. These test cases represent the priority we decided to give the individual feature selection objectives for the purposes of this study. Each test case has a set of requirements to be satisfied to pass:  
* _Test Case A_. One irrelevant feature is selected, e.g., \(F\)4. This case should have a fitness lower than all other test cases since no relevant features are selected. Performance is therefore expected to be poor. * _Test Case B_. All features are selected. This case should have a fitness greater than test case A. A solution with discriminatory information should perform better than a solution with no discriminatory information. \begin{table}
\begin{tabular}{c c} \hline \hline
**Feature** & **Description** \\ \hline F1 & A unique incremental ID \\ F2 & A completely correlated feature to the class \\ F3 & A completely correlated and a completely redundant feature \\ F4 & A completely irrelevant feature \\ Class & true or false \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Artificial dataset features. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**F1** & **F2** & **F3** & **F4** & **Class** \\ \hline
1 & 1 & 2 & 0 & true \\
2 & 2 & 1 & 0 & false \\
3 & 1 & 2 & 0 & true \\
4 & 2 & 1 & 0 & false \\
5 & 1 & 2 & 0 & true \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Robustness artificial dataset excerpt. * _Test Case C_. One relevant feature is selected, e.g., \(F\)2 or \(F\)3. This case should result in the highest fitness over all other test cases. This is the ideal solution, where a single feature can perfectly classify. * _Test Case D_. Two relevant features, but one redundant, are selected, e.g., \(F\)2 and \(F\)3. This case should have a fitness greater than test case A, but lower than test case C. The features are both correlated; therefore, it is expected to perform better than test case A where there are no relevant features. Redundant features are included; therefore, performance should be worse than the best case scenario in test case C.
* _Test Case E_. One irrelevant feature and one relevant feature are selected, e.g., \(F\)2 and \(F\)4. This case should have a fitness greater than test case B and D but lower than test case C. This case should be better than test case B because it uses fewer features. This case contains a feature that is strongly correlated with the class and should therefore perform better than a solution with no relevant features as in test case A. One additional feature is considered than for test case C; therefore, it should be penalised more. * _Test Case F_. A mix of one relevant, one redundant and one irrelevant feature, e.g., \(F\)1, \(F\)2 and \(F\)4. This case should have a fitness better than test case B and worse than test case E. One feature was removed in this test case in comparison to test case B. Therefore, this test case is expected to perform better than test case B.