# SPL Features Quantification and Selection Based on Multiple Multi-Level Objectives
## 6 System Evaluation and Results  
We evaluate the three Feature Selection Algorithms (FSAs), proposed in Section 4, in terms of product added value, execution time and number of selected features. The FSAs were applied on a dataset extracted from the Unity3D assets store [17], developed as part of this research. The experiments were executed in a custom-made tool called Feature Selection Engine (FSE) [16], also developed as part of this research. The experiment was conducted as follows; The original dataset was divided  
Figure 5: FSE main window. Figure 6: FSE setting feature level objectives. to 8 datasets [available in Supplementary file], containing 10, 20, 100, 1000, 6174, 10,000, 20,000, and 40,611 features. The raw datasets were imported to the FSE, where all features were quantified based on eight user-defined objectives. Three different user-defined product profiles, containing different product and category level objectives were set, in order to evaluate the algorithms. The FSE selected features based on the product profiles and the datasets containing the quantified features. Evaluation of the dataset was performed using three case studies. 1. Case Study-1: In this scenario, two product level constraints have been set: The total property value for the field \"finalPrice\" must be less than 500 and the total property value for the field \"Size\" must be less than 60,000. 2. Case Study-2: In this scenario, one product level constraint has been set: The total property value for the field \"finalPrice\" must be less than 100. 3. Case Study-3: In this scenario, three product level constraints have been set: The total property value for the field \"finalPrice\" must be less than 1000, the total property value for the field \"numberOfVotes\" must be greater than 5, and the total property value for the field \"Size\" must be less than 30,000. Along with these product level constraints, there is a constrain on the number of occurrences of a specific category, i.e., a category can appear only twice. With all these three case studies, various parameters like time (in milliseconds), added value, and features added were obtained for the three feature selection algorithms on different datasets. Their outcome is averaged and provided in Table 7.  
Table 7 is depicted in Figures 7-9. In Table 7, the number of features provided in the first column indicates the eight datasets generated for evaluation purposes. In Figures 7-9, it could be observed that the performance among the Falcon and Jaguar are close to each other due to presence of limited dataset. The Snail algorithm could be applied only for the first two datasets and thus not shown in the Figures 7-9. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Algorithm (Features)** & **Time (Millisecond)** & **Added value** & **Features Added** \\ \hline Falcon (10) & 5.33 & 1150.12 & 4.33 \\ \hline Jaguar (10) & 4.67 & 1150.12 & 4.33 \\ \hline Snail (10) & 336 & 1150.12 & 4.33 \\ \hline Falcon (20) & 9 & 2061.03 & 7 \\ \hline Jaguar (20) & 8.67 & 2057.64 & 7 \\ \hline Snail (20) & 742,654.67 & 2061.03 & 7 \\ \hline Falcon (100) & 37.33 & 6538.64 & 21 \\ \hline Jaguar (100) & 35 & 5983.48 & 20.33 \\ \hline Falcon (1000) & 1163 & 24511.07 & 66 \\ \hline Jaguar (1000) & 430 & 30,331.39 & 86.67 \\ \hline Falcon (6174) & 3080.67 & 55,372.52 & 184.33 \\ \hline Jaguar (6174) & 5501 & 54,555.97 & 185.33 \\ \hline Falcon (10,000) & 5845 & 84,594.1 & 291.33 \\ \hline Jaguar (10,000) & 11,294.67 & 98,101.64 & 342.33 \\ \hline Falcon (20,000) & 19,238.67 & 139,671.84 & 511.67 \\ \hline Jaguar (20,000) & 47,739.67 & 155,199.57 & 560.67 \\ \hline Falcon (40,611) & 85,979.33 & 384,726.32 & 1383.67 \\ \hline Jaguar (40,611) & 226,166 & 448,012.86 & 1606.33 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Average comparison between feature selection algorithms. Figure 8: Average added value. Figure 7: Average time in milliseconds. Observation from these figures reveal that Jaguar outperforms Falcon in terms of added value and features added.