Package: target-permutation-importances
Target Permutation Importances (Null Importances)[Source][Bug Report][Documentation][API Reference]OverviewNull Importances is normalized feature importance measures that can correct the feature importance bias.
The method is based on repeated permutations of the outcome
vector for estimating the distribution of measured importance
for each variable in a non-informative setting.Overall, this packageFit the given model class $M$ times with different model'srandom_stateto get $M$ actual feature importances of feature f: $A_f = [a_{f_1},a_{f_2}...a_{f_M}]$.Fit the given model class with different model'srandom_stateandshuffled targetsfor $N$ times to get $N$ feature random importances: $R_f = [r_{f_1},r_{f_2}...r_{f_N}]$.Compute the final importances of a feature $f$ by various methods, such as:$I_f = Avg(A_f) - Avg(R_f)$$I_f = Avg(A_f) / (Avg(R_f) + 1)$$I_f = Wasserstein Distance(A_f, R_f)$We want $M \ge 1$ and $N \gg 1$. Having $M=1$ means the actual importances depends on only 1 model'srandom_state(Which is also fine).Not to be confused withsklearn.inspection.permutation_importance,
this sklearn method is about feature permutation instead of target permutation.This method were originally proposed/implemented by:[Paper] Permutation importance: a corrected feature importance measure[Kaggle Notebook] Feature Selection with Null ImportancesFeaturesCompute null importances with only one function call.Support models that provide information about feature importance (e.g. coef_, feature_importances_), such asRandomForestClassifier,RandomForestRegressor,XGBClassifier.XGBRegressor,LGBMClassifier,LGBMRegressor,CatBoostClassifier,CatBoostRegressor,Lasso,LinearSVC, etc.Supportsklearn'sMultiOutputClassifierorMultiOutputRegressorinterface.Support data inpandas.DataFrameandnumpy.ndarrayHighly customizable with both the exposedcomputeandgeneric_computefunctions.Proven effectiveness in Kaggle competitions and inOur Benchmarks Results.BenchmarksHere are some examples of Top Kaggle solutions using this method:YearCompetitionMedalLink2023Predict Student Performance from Game PlayGold3rd place solution2019Elo Merchant Category RecommendationGold16th place solution2018Home Credit Default RiskGold10th place solutionBelow show the benchmark results of running null-importances with feature selection on multiple models and datasets. "better" means it is better than running feature selection with the model's built-in feature importances. Details can be seenhere.modeln_datasetn_betterbetter %RandomForestClassifier1010100.0RandomForestRegressor12866.67XGBClassifier10770.0XGBRegressor12758.33LGBMClassifier10880.0LGBMRegressor12866.67CatBoostClassifier10660.0CatBoostRegressor12866.67Installationpip install target-permutation-importancesor with poetry:poetry add target-permutation-importancesIf with Python 3.8.x & Pandas 1.x.x:pip install target-permutation-importances==1.19.0or with poetry:poetry add target-permutation-importances==1.19.0Although this package is tested on models fromsklearn,xgboost,catboost,lightgbm, they are not
a hard requirement for the installation, you can use this package for any model if it implements thesklearninterface.
For models that don't followsklearninterface, you can use the exposedgeneric_computemethod as discussed in the
Advance Usage / Customization section.Dependencies:[tool.poetry.dependencies]
python = "^3.9"
numpy = "^1.23.5"
pandas = "^1.5.3"
tqdm = "^4.48.2"
beartype = "^0.14.1"
scipy = "^1.9"Get Started (Functional APIs)Tree Models withfeature_importances_Attribute# import the packageimporttarget_permutation_importancesastpi# Prepare a datasetimportpandasaspdfromsklearn.datasetsimportload_breast_cancer# Modelsfromsklearn.ensembleimportRandomForestClassifierdata=load_breast_cancer()# Convert to a pandas dataframeXpd=pd.DataFrame(data.data,columns=data.feature_names)# Compute permutation importances with default settingsresult_df=tpi.compute(model_cls=RandomForestClassifier,# The constructor/class of the model.model_cls_params={# The parameters to pass to the model constructor. Update this based on your needs."n_jobs":-1,},model_fit_params={},# The parameters to pass to the model fit method. Update this based on your needs.X=Xpd,# pd.DataFrame, np.ndarrayy=data.target,# pd.Series, np.ndarraynum_actual_runs=2,num_random_runs=10,# Options: {compute_permutation_importance_by_subtraction, compute_permutation_importance_by_division}# Or use your own function to calculate.permutation_importance_calculator=tpi.compute_permutation_importance_by_subtraction,)print(result_df[["feature","importance"]].sort_values("importance",ascending=False).head())Fork above code fromKaggle.Outputs:Running 2 actual runs and 10 random runs
100%|██████████| 2/2 [00:01<00:00,  1.62it/s]
100%|██████████| 10/10 [00:06<00:00,  1.46it/s]
                 feature  importance
25       worst perimeter    0.117495
22  worst concave points    0.089949
26          worst radius    0.084632
7    mean concave points    0.064289
20            worst area    0.062485
8         mean concavity    0.047122
10        mean perimeter    0.029270
5              mean area    0.014566
11           mean radius    0.014346
0             area error    0.000693Linear Models withcoef_Attribute# import the packageimporttarget_permutation_importancesastpi# Prepare a datasetimportpandasaspdfromsklearn.datasetsimportload_breast_cancer# Modelsfromsklearn.svmimportLinearSVCdata=load_breast_cancer()# Convert to a pandas dataframeXpd=pd.DataFrame(data.data,columns=data.feature_names)# Compute permutation importances with default settingsresult_df=tpi.compute(model_cls=LinearSVC,# The constructor/class of the model.model_cls_params={"max_iter":1000},# The parameters to pass to the model constructor. Update this based on your needs.model_fit_params={},# The parameters to pass to the model fit method. Update this based on your needs.X=Xpd,# pd.DataFrame, np.ndarrayy=data.target,# pd.Series, np.ndarraynum_actual_runs=1,num_random_runs=10,# Options: {compute_permutation_importance_by_subtraction, compute_permutation_importance_by_division}# Or use your own function to calculate.permutation_importance_calculator=tpi.compute_permutation_importance_by_subtraction,)print(result_df[["feature","importance"]].sort_values("importance",ascending=False).head(10))Fork above code fromKaggle.Outputs:feature  importance
10     mean perimeter    0.067352
29      worst texture    0.029602
11        mean radius    0.029509
26       worst radius    0.026499
21  worst compactness    0.010139
23    worst concavity    0.009149
25    worst perimeter    0.008779
14       mean texture    0.007845
0          area error    0.007540
20         worst area    0.004508Withsklearn.multioutput# import the packageimporttarget_permutation_importancesastpi# Prepare a datasetimportpandasaspdfromsklearn.datasetsimportmake_regression# Modelsfromsklearn.ensembleimportRandomForestRegressorfromsklearn.multioutputimportMultiOutputClassifier,MultiOutputRegressor# Multi target regressionX,y=make_regression(n_samples=100,n_features=20,n_targets=5,)# Compute permutation importances with default settingsresult_df=tpi.compute(model_cls=MultiOutputRegressor,model_cls_params={# The constructor/class of MultiOutputRegressor"estimator":RandomForestRegressor(n_estimators=2),},model_fit_params={},# The parameters to pass to the model fit method. Update this based on your needs.X=X,# pd.DataFrame, np.ndarrayy=y,# pd.Series, np.ndarraynum_actual_runs=2,num_random_runs=10,# Options: {compute_permutation_importance_by_subtraction, compute_permutation_importance_by_division}# Or use your own function to calculate.permutation_importance_calculator=tpi.compute_permutation_importance_by_subtraction,)print(result_df[["feature","importance"]].sort_values("importance",ascending=False).head())Fork above code fromKaggle.Outputs:Running 2 actual runs and 10 random runs
100%|██████████| 2/2 [00:00<00:00,  4.97it/s]
100%|██████████| 10/10 [00:01<00:00,  5.10it/s]
    feature  importance
19       19    0.233211
10       10    0.058238
0         0    0.049782
17       17    0.031321
2         2    0.028209You can find more detailed examples in the "Feature Selection Examples" section.CustomizationChanging model or parametersYou can pick your own model by changingmodel_cls,model_cls_paramsandmodel_fit_params, for example, using withLGBMClassifierwith aimportance_type=gainandcolsample_bytree=0.5:result_df=tpi.compute(model_cls=LGBMClassifier,# The constructor/class of the model.model_cls_params={# The parameters to pass to the model constructor. Update this based on your needs."n_jobs":-1,"importance_type":"gain","colsample_bytree":0.5,},model_fit_params={},# The parameters to pass to the model fit method. Update this based on your needs.X=Xpd,# pd.DataFrame, np.ndarrayy=data.target,# pd.Series, np.ndarraynum_actual_runs=2,num_random_runs=10,# Options: {compute_permutation_importance_by_subtraction, compute_permutation_importance_by_division}# Or use your own function to calculate.permutation_importance_calculator=tpi.compute_permutation_importance_by_subtraction,)Note: Tree models are greedy. Usually it is a good idea to introduce some randomness into the tree model by settingcolsample_*params.
It forces the model to explore the importances of different features. In other words, setting these
parameters avoid a feature from being under-representative in the importance calculation because of having another highly correlated feature.Changing null importances calculationYou can pick your own calculation method by changingpermutation_importance_calculator.
There are 3 provided calculations:tpi.compute_permutation_importance_by_subtractiontpi.compute_permutation_importance_by_divisiontpi.compute_permutation_importance_by_wasserstein_distanceYou can also implement you own calculation function and pass it in. The function needs to followPermutationImportanceCalculatorTypespecification, you can find it inAPI ReferenceAdvance CustomizationThis package exposestpi.generic_computeto allow advance customization.
Read the followings for details:generic_compute API referencetarget_permutation_importances.functional.py.Get Started (scikit-learn APIs)TargetPermutationImportancesWrapperfollows scikit-learn interfaces and support scikit-learn feature selection method such asSelectFromModel:# Import the packageimporttarget_permutation_importancesastpi# Prepare a datasetimportpandasaspdimportnumpyasnpfromsklearn.datasetsimportload_breast_cancer# Modelsfromsklearn.feature_selectionimportSelectFromModelfromsklearn.ensembleimportRandomForestClassifierdata=load_breast_cancer()# Convert to a pandas dataframeXpd=pd.DataFrame(data.data,columns=data.feature_names)# Compute permutation importances with default settingswrapped_model=tpi.TargetPermutationImportancesWrapper(model_cls=RandomForestClassifier,# The constructor/class of the model.model_cls_params={# The parameters to pass to the model constructor. Update this based on your needs."n_jobs":-1,},num_actual_runs=2,num_random_runs=10,# Options: {compute_permutation_importance_by_subtraction, compute_permutation_importance_by_division}# Or use your own function to calculate.permutation_importance_calculator=tpi.compute_permutation_importance_by_subtraction,)wrapped_model.fit(X=Xpd,# pd.DataFrame, np.ndarrayy=data.target,# pd.Series, np.ndarray# And other fit parameters for the model.)# Get the feature importances as a pandas dataframeresult_df=wrapped_model.feature_importances_dfprint(result_df[["feature","importance"]].sort_values("importance",ascending=False).head())# Select top-5 features with sklearn `SelectFromModel`selector=SelectFromModel(estimator=wrapped_model,prefit=True,max_features=5,threshold=-np.inf).fit(Xpd,data.target)selected_x=selector.transform(Xpd)print(selected_x.shape)print(selector.get_feature_names_out())Fork above code fromKaggle.Outputs:Running 2 actual runs and 10 random runs
100%|██████████| 2/2 [00:01<00:00,  1.80it/s]
100%|██████████| 10/10 [00:06<00:00,  1.55it/s]
                 feature  importance
22       worst perimeter    0.151953
27  worst concave points    0.124407
20          worst radius    0.119090
7    mean concave points    0.098747
23            worst area    0.096943
(569, 5)
['mean concave points' 'worst radius' 'worst perimeter' 'worst area'
 'worst concave points']Feature Selection ExamplesFeature Selection for Binary Classification with RandomForestClassifierFeature Selection for Binary Classification with XGBClassifierDevelopment Setup and Contribution GuidePython VersionYou can find the suggested development Python version in.python-version.
You might consider setting upPyenvif you want to have multiple Python versions on your machine.Python packagesThis repository is setup withPoetry. If you are not familiar with Poetry, you can find package requirements listed inpyproject.toml.
Otherwise, you can just set it up withpoetry installRun BenchmarksTo run the benchmark locally on your machine, runmake run_tabular_benchmarkorpython -m benchmarks.run_tabular_benchmarkMake ChangesFollowing theMake Changes Guide from GithubBefore committing or merging, please run the linters defined inmake lintand the tests defined inmake test