# Framework for the Ensemble of Feature Selection Methods
## 3 Results
### Implementation of the Conceptual Framework  
The implementation of the conceptual framework was developed to validate the proposal described in Figure 2. The Scikit-learn Machine Learning Library [35] was used to develop a tool that represents the framework. The solution selects features by different FS algorithms and then aggregates their outputs through consensus. The framework developed allows (i) to read, fix, and impute the values of datasets, (ii) to remove dataset features with high correlation, low variance, or null values, (iii) to generate \(n\) subsets of relevant features using \(n\) FS algorithms, (iv) to aggregate the subsets generated using methods based on voting, and (v) to evaluate the performance of the subset of features generated by our EFS.  
Considering the above, Figure 3 describes the implementation of the framework of Figure 2. The solution groups the functions and methods in packages according to their objective.  
* Interface Module: It describes a class. The module exposes the functionalities of the framework to new implementations.
* EFS Module: It is the core of the framework. This component includes all the functionalities associated with the FS based on our ensemble method.  
Figure 3: Developed Framework.  
* Evaluation Package: This groups a set of functions to evaluate the accuracy and stability of an EFS output.
* Selection Package: It contains a set of methods to select subsets of relevant features.
* Aggregation Package: It integrates the outputs of n methods of FS using a criterion to aggregate the outputs.
* SCIKIT-LEARN: It is a Machine Learning library that supports the implementation of our EFS package.
* Data Module: It includes the functions to read and preprocess the datasets. The module allows the reading of data from CVS files to adjust them according to assumptions and constraints considered in the conceptual framework design.
* Offset Package: It describes functionalities to calculate an offset dataset to avoid negative values.
* Imputation Package: It implements the Multivariate Imputation to handle datasets with missing values. The authors presented the implementation and evaluation of this component in a previous study [36].
* Outliers Detection Package: This package considers a software component to implement methods of detection of outliers. These methods include, for instance, basic methods as the Standard Deviation or advanced methods as Novelty and Outlier Detection from scikit-learn [37].  
It is essential to mention that the constraints defined by the conceptual proposal were solved in the implementation framework with the development of the offset, imputation, and outlier detection packages. However, the framework does not support all data preparation tasks or feature engineering (extraction and transformation of data) with these packages. Therefore, the goal of the packages is only to support common problems associated with the constraints of the FS algorithms.  
Additionally, this framework describes generic phases and algorithms that support ensemble feature selection. However, the data science team should adjust hyperparameters for the algorithms considering their data. A detailed example of how to use the EFS framework is described in a previous study developed by the authors to evaluate the EFS quality [38].