# Feature Selection Algorithms as One of the Python Data Analytical Tools +
## 6 ITMO FS Library Usage Examples and Performance Tests  
For all code samples, imports from the Listing 2 are required. ```
1 importtime
2 fromskfeature.function.statistical_basedimportgini_index
3 fromsklearn.datasetsimportload_iris
4 fromsklearn.feature_selectionimportSelectKBest
5 fromsklearn.linear_modelimportLogisticRegression
6 fromsklearn.metricsimportfl_score
7 fromsklearn.svmimportSVC
8 fromITMO_FS.filtersimport*
9 fromITMO_FS.hybrid.MelifimportMelf
10 fromITMO_FS.wrappers.AddDelWrapperimport*
11 fromITMO_FS.wrappers.BackwardSelectionimport*
```  
Listing 2: Imports for this section experimental setup. During testing, we used the following software versions: sklearn 0.21.3, MLFeatureSelection 0.0.9.5.1, skrebate 0.6, python 3.6.9, numpy 1.17.3, scipy 1.3.1, Boruta 0.3, and ReliefF 0.1.2. In the test_filter function, an example of the filter class with Spearman correlation and \"Best by value\" cutting rule that equals 0.99 is shown. This notation means that all features which have score higher than 0.99 will be selected. A possibility to customize the cutting rule such as in this example is only supported by the ITMO FS library and is shown in Listing 3 For this example, iris [30] dataset was chosen. ```
1 deftest_filter(self):
2 filtering=UnivariateFilter(spearman_corr,select_best_by_value(0.99))
3 data,target=load_iris(True)
4 res=filtering.run(data,target)
5 print(data.shape,'---->',res.shape)
```  
Listing 3: Filter usage on the iris dataset with the custom cutting rule. In Listing 4, an example of UnivariateFilter class with Pearson correlation and \"Best by value\" cutting rule that equals 0.0 is shown. A possibility to use lambda functions for cutting rules is only supported by the ITMO FS library. This example uses orlraws10P dataset [31]. ```
1 deftest_filter(self):
2 value=0
3 rule=lambdscores:[kfork,vinscores.items()ifv>value]
4 filtering=UnivariateFilter(pearson_corr,rule)
5 data=scipy.io.loadmat(orlaws)
6 data,target=data['X'],data['Y']
7 res=filtering.run(data,target)
8 print(data.shape,'->',res.shape)
```  
Listing 5: Comparison of the K-best cutting rule and Pearson correlation usage at ITMO FS and scikit libraries. In Listing 6, the AddDel (Line 4) wrapper method with basic logistic regression (Line 3) as estimator is shown. For quality measure function, \(F_{1}\) score was chosen. Test dataset here is Basehock. ```
1 deftest_add_del():
2 data,target=basehock['X'],basehock['Y']
3 lr=LogisticRegression()
4 wrapper=Add_del(lr,fl_score)
5 wrapper.fit(data,target)
6 print(wrapper.best_score)
```  
Listing 6: Example of Add-del wrapper usage. In Listing 7, the backward selection wrapper (Line 4) method with logistic regression (Line 3) as estimator is shown. Test dataset here is Basehock. Although the BackwardSelection function is also a wrapper, it has an additional parameter \"100\" that defines maximum number of features to be removed. In this case, it is a \"stopping criteria\" in the standard wrapper notation. ```
1 deftest_backward_selection():
2 data,target=basehock['X'],basehock['Y']
3 lr=LogisticRegression()
4 wrapper=BackwardSelection(lr,100,gini_index)
5 wrapper.fit(data,target)
6 print(wrapper.best_score)
```  
Listing 8: Example of ensemble algorithm MeLiF usage. Listing 8 shows an example of the MeLiF class with Support vector classifier (Line 7) as estimator Chosen filters are Gini index (Line 4), ratio (Line 5), and information gain (Line 6). \(F_{1}\) score was chosen as quality measure for MeLiF (Line 7). All input parameters of the ensemble including the cutting rules can be tuned. ```
1 deftest_melf():
2 data,target=basehock['X'],.basehock['Y']
3_filters=[
4 UnivariateFilter(gini_index,select_best_by_value(0.4)),
5 UnivariateFilter(F_ratio(data.shape[1]),select_best_by_value(0.6)),
6 UnivariateFilter(ig_measure,select_best_by_value(-0.4))]
7 melif=Melf(_filters,fl_score)
8 melif.fit(data,target)
9 estimator=SVC()
10 melif.run(select_k_best(50),estimator)
```  
Listing 8: Example of ensemble algorithm MeLiF usage.