# Feature Selection Algorithms as One of the Python Data Analytical Tools +
## 5 ITMO FS library Architecture and Comparison  
The main issues observed in the libraries presented in Sections 3 and 4 libraries are as follows:  
* does not support customizable cutting rules for filters (nearly all of them);
* does not support ensemble or hybrid feature selection algorithms (which are actually the most commonly used in recent years, nearly all of them);
* has high requirements and long delays for new algorithms to be added (scikit-learn);
* uses input data paradigm as basis for architecture, as described in Section 2.3, which is not usually considered (ASU); and
* programmed in other languages so has to be either adopted or wrapped with additional computational costs. Thus, the ITMO FS library was decided to be developed. In Figure 1, the ITMO FS library architecture is shown. It contains filters, wrappers, hybrid, and embedded parts, each of which represents a set of methods from traditional classification. Unfortunately, ensembles are not currently presented in the library scheme as they are still under development. This categorization was inspired by traditional feature selection categorization. All filters are generally described as shown in Section 2.1, thus they need to have some feature quality measure and a cutting rule. The wrapper module contains several wrapper algorithms. Each of them takes an estimator as input parameter and improves the feature set regarding model performance. The embedded part contains MOSS, MOSNS, and RFE algorithms, each of which selects the feature set according to its importance to the model. The hybrid module for now contains only MeLiF algorithm that was developed by ITMO University [14]. Figure 1: ITMO FS library architecture. The only prerequisite for library is numpy package; the library is developed with Python 3 language. The current version of the library contents is shown below and also it could be found on the architecture scheme in Figure 1:  
* UnivariateFilter--class for constructing custom filter (UnivariateFilter.py);
* MRMR--class for specific filter (MRMRFilter.py);
* VDM--class for specific filter (VDM.py);  
* MeLiF--class for basic MeLiF (MeLiF.py);  
* AddDel--class for ADD-DEL wrapper (AddDelWrapper.py);
* Backward elimination--class for backward elimination wrapper (BackwardSelection.py);
* Sequential forward selection--class for sequential forward selection (SequentialForwardSelection.py);  
* MOSNS--Minimizing Overlapping Selection under No-Sampling (MOSNS.py);
* MOSS--Minimizing Overlapping Selection under SMOTE (MOSS.py);
* Recursive Feature Elemenation--class for RFE method (RecursiveElimination.py). Spearman correlation, Pearson correlation, information gain, Gini index, F-ratio, fit criterion, and symmetric uncertainty are custom measures that are stored with cutting rules in the Filter file. In the future, when we add more basic measures to the library and some more exotic cutting rules, they could be easily added to the library through this file. MeLiF is one of the hybrid methods and it aggregates the result of filter measures and wraps an estimator, greedy optimizing their ensemble weights. The pipeline is displayed in Figure 2. This algorithm builds a linear combination of the input filters quality measures with some \(\alpha\) coefficients and then tunes them in the optimization cycle of the model \(C\) with some quality measure \(Q_{C}\) (for example, \(F_{1}\) score). In this case, a single cutting rule \(\kappa\) is applied for the whole combination. Figure 2: Hybrid algorithm MeLiF pipeline. As shown in Table 1, the Arizona State University feature selection library has the most implemented algorithms in comparison with others, but this library was updated last time about two years ago. Moreover, it does not have some basic feature selection algorithms such as information gain filter, AddDel, RFE, ROC filter, forward and backward selection, and some others. Scikit-learn feature selection module seems to have some rarely used methods and the general volume of presented algorithms is rather small: it has only six of them.