Package: selective
Selective: Feature Selection LibrarySelectiveis a white-box feature selection library that supports supervised and unsupervised selection methods for classification and regression tasks.Selective also provides optimized item selection based on diversity of text embeddings (viaTextWiser) and
the coverage of binary labels by solving a multi-objective optimization problem (CPAIOR'21,DSO@IJCAI'22). The approach showed to speed-up online experimentation significantly and boost recommender systemsNVIDIA GTC'22.The library provides:Simple to complex selection methods: Variance, Correlation, Statistical, Linear, Tree-based, or Customized.Text-based selectionto maximize diversity in text embeddings and metadata coverage.Interoperable with data frames as the input.Automated task detection. No need to know what feature selection method works with what machine learning task.Benchmarking multiple selectors using cross-validation with built-in parallelization.Inspection of the results and feature importance.Selective is developed by the Artificial Intelligence Center of Excellence at Fidelity Investments.Quick Start# Import Selective and SelectionMethodfromsklearn.datasetsimportfetch_california_housingfromfeature.utilsimportget_data_labelfromfeature.selectorimportSelective,SelectionMethod# Datadata,label=get_data_label(fetch_california_housing())# Feature selectors from simple to more complexselector=Selective(SelectionMethod.Variance(threshold=0.0))selector=Selective(SelectionMethod.Correlation(threshold=0.5,method="pearson"))selector=Selective(SelectionMethod.Statistical(num_features=3,method="anova"))selector=Selective(SelectionMethod.Linear(num_features=3,regularization="none"))selector=Selective(SelectionMethod.TreeBased(num_features=3))# Feature reductionsubset=selector.fit_transform(data,label)print("Reduction:",list(subset.columns))print("Scores:",list(selector.get_absolute_scores()))Available MethodsMethodOptionsVariance per FeaturethresholdCorrelation pairwise FeaturesPearson Correlation CoefficientKendall Rank Correlation CoefficientSpearman's Rank Correlation CoefficientStatistical AnalysisANOVA F-test ClassificationF-value RegressionChi-SquareMutual Information ClassificationVariance Inflation FactorLinear MethodsLinear RegressionLogistic RegressionLasso RegularizationRidge RegularizationTree-based MethodsDecision TreeRandom ForestExtra Trees ClassifierXGBoostLightGBMAdaBoostCatBoostGradient Boosting TreeText-based Methodsfeaturization_method=TextWiseroptimization_method = ["exact", "greedy", "kmeans", "random"]cost_metric = ["unicost", "diverse"]Benchmarking# Importsfromsklearn.datasetsimportfetch_california_housingfromfeature.utilsimportget_data_labelfromxgboostimportXGBClassifier,XGBRegressorfromfeature.selectorimportSelectionMethod,benchmark,calculate_statistics# Datadata,label=get_data_label(fetch_california_housing())# Selectorscorr_threshold=0.5num_features=3tree_params={"n_estimators":50,"max_depth":5,"random_state":111,"n_jobs":4}selectors={# Correlation methods"corr_pearson":SelectionMethod.Correlation(corr_threshold,method="pearson"),"corr_kendall":SelectionMethod.Correlation(corr_threshold,method="kendall"),"corr_spearman":SelectionMethod.Correlation(corr_threshold,method="spearman"),# Statistical methods"stat_anova":SelectionMethod.Statistical(num_features,method="anova"),"stat_chi_square":SelectionMethod.Statistical(num_features,method="chi_square"),"stat_mutual_info":SelectionMethod.Statistical(num_features,method="mutual_info"),# Linear methods"linear":SelectionMethod.Linear(num_features,regularization="none"),"lasso":SelectionMethod.Linear(num_features,regularization="lasso",alpha=1000),"ridge":SelectionMethod.Linear(num_features,regularization="ridge",alpha=1000),# Non-linear tree-based methods"random_forest":SelectionMethod.TreeBased(num_features),"xgboost_classif":SelectionMethod.TreeBased(num_features,estimator=XGBClassifier(**tree_params)),"xgboost_regress":SelectionMethod.TreeBased(num_features,estimator=XGBRegressor(**tree_params))}# Benchmark (sequential)score_df,selected_df,runtime_df=benchmark(selectors,data,label,cv=5)print(score_df,"\n\n",selected_df,"\n\n",runtime_df)# Benchmark (in parallel)score_df,selected_df,runtime_df=benchmark(selectors,data,label,cv=5,n_jobs=4)print(score_df,"\n\n",selected_df,"\n\n",runtime_df)# Get benchmark statistics by featurestats_df=calculate_statistics(score_df,selected_df)print(stats_df)Text-based SelectionThis example shows how to use text-based selection. In this scenario, we would like to select a subset of articles that is most diverse in the text embedding space and covers a range of topics.# Import Selective and TextWiserimportpandasaspdfromfeature.selectorimportSelective,SelectionMethodfromtextwiserimportTextWiser,Embedding,Transformation# Data with the text content of each articledata=pd.DataFrame({"article_1":["article text here"],"article_2":["article text here"],"article_3":["article text here"],"article_4":["article text here"],"article_5":["article text here"]})# Labels to denote 0/1 coverage metadata for each article# across four labels, e.g., sports, international, entertainment, sciencelabels=pd.DataFrame({"article_1":[1,1,0,1],"article_2":[0,1,0,0],"article_3":[0,0,1,0],"article_4":[0,0,1,1],"article_5":[1,1,1,0]},index=["label_1","label_2","label_3","label_4"])# TextWiser featurization method to create text embeddingstextwiser=TextWiser(Embedding.TfIdf(),Transformation.NMF(n_components=20))# Text-based selection# The goal is to select a subset of articles# that is most diverse in the text embedding space of articles# and covers the most labels in each topicselector=Selective(SelectionMethod.TextBased(num_features=2,featurization_method=textwiser))# Feature reductionsubset=selector.fit_transform(data,labels)print("Reduction:",list(subset.columns))Visualizationimportpandasaspdfromsklearn.datasetsimportfetch_california_housingfromfeature.utilsimportget_data_labelfromfeature.selectorimportSelectionMethod,Selective,plot_importance# Datadata,label=get_data_label(fetch_california_housing())# Feature Selectorselector=Selective(SelectionMethod.Linear(num_features=8,regularization="none"))subset=selector.fit_transform(data,label)# Plot Feature Importancedf=pd.DataFrame(selector.get_absolute_scores(),index=data.columns)plot_importance(df)InstallationSelective requiresPython 3.7+and can be installed from PyPI usingpip install selective.SourceAlternatively, you can build a wheel package on your platform from scratch using the source code:gitclonehttps://github.com/fidelity/selective.gitcdselective
pipinstallsetuptoolswheel# if wheel is not installedpythonsetup.pysdistbdist_wheel
pipinstalldist/selective-X.X.X-py3-none-any.whlTest your setupcd selective
python -m unittest discover testsSupportPlease submit bug reports and feature requests asIssues.LicenseSelective is licensed under theGNU GPL 3.0.