Package: vimpy
Python/vimpy: inference on algorithm-agnostic variable importanceSoftware author:Brian WilliamsonMethodology authors:Brian Williamson,Peter Gilbert,Noah Simon,Marco CaroneR package:https://github.com/bdwilliamson/vimpIntroductionIn predictive modeling applications, it is often of interest to determine the relative contribution of subsets of features in explaining an outcome; this is often called variable importance. It is useful to consider variable importance as a function of the unknown, underlying data-generating mechanism rather than the specific predictive algorithm used to fit the data. This package provides functions that, given fitted values from predictive algorithms, compute nonparametric estimates of variable importance based on $R^2$, deviance, classification accuracy, and area under the receiver operating characteristic curve, along with asymptotically valid confidence intervals for the true importance.For more details, please see the accompanying manuscripts "Nonparametric variable importance assessment using machine learning techniques" by Williamson, Gilbert, Carone, and Simon (Biometrics, 2020),"A unified approach for inference on algorithm-agnostic variable importance"by Williamson, Gilbert, Simon, and Carone (arXiv, 2020), and"Efficient nonparametric statistical inference on population feature importance using Shapley values"by Williamson and Feng (arXiv, 2020; to appear in the Proceedings of the Thirty-seventh International Conference on Machine Learning [ICML 2020]).InstallationYou may install a stable release ofvimpyusingpipby runningpython pip install vimpyfrom a Terminal window. Alternatively, you may install within avirtualenvenvironment.You may install the current dev release ofvimpyby downloading this repository directly.IssuesIf you encounter any bugs or have any specific feature requests, pleasefile an issue.ExampleThis example shows how to usevimpyin a simple setting with simulated data and using a single regression function. For more examples and detailed explanation, please see theRvignette.## load required librariesimportnumpyasnpimportvimpyfromsklearn.ensembleimportGradientBoostingRegressorfromsklearn.model_selectionimportGridSearchCV## -------------------------------------------------------------## problem setup## -------------------------------------------------------------## define a function for the conditional mean of Y given Xdefcond_mean(x=None):f1=np.where(np.logical_and(-2<=x[:,0],x[:,0]<2),np.floor(x[:,0]),0)f2=np.where(x[:,1]<=0,1,0)f3=np.where(x[:,2]>0,1,0)f6=np.absolute(x[:,5]/4)**3f7=np.absolute(x[:,6]/4)**5f11=(7./3)*np.cos(x[:,10]/2)ret=f1+f2+f3+f6+f7+f11returnret## create datanp.random.seed(4747)n=100p=15s=1# importance desired for X_1x=np.zeros((n,p))foriinrange(0,x.shape[1]):x[:,i]=np.random.normal(0,2,n)y=cond_mean(x)+np.random.normal(0,1,n)## -------------------------------------------------------------## preliminary step: get regression estimators## -------------------------------------------------------------## use grid search to get optimal number of trees and learning ratentrees=np.arange(100,500,100)lr=np.arange(.01,.1,.05)param_grid=[{'n_estimators':ntrees,'learning_rate':lr}]## set up cv objectscv_full=GridSearchCV(GradientBoostingRegressor(loss='ls',max_depth=1),param_grid=param_grid,cv=5)cv_small=GridSearchCV(GradientBoostingRegressor(loss='ls',max_depth=1),param_grid=param_grid,cv=5)## -------------------------------------------------------------## get variable importance estimates## -------------------------------------------------------------# set seednp.random.seed(12345)## set up the vimp objectvimp=vimpy.vim(y=y,x=x,s=1,pred_func=cv_full,measure_type="r_squared")## get the point estimate of variable importancevimp.get_point_est()## get the influence function estimatevimp.get_influence_function()## get a standard errorvimp.get_se()## get a confidence intervalvimp.get_ci()## do a hypothesis test, compute p-valuevimp.hypothesis_test(alpha=0.05,delta=0)## display the estimates, etc.vimp.vimp_vimp.se_vimp.ci_vimp.p_value_vimp.hyp_test_## -------------------------------------------------------------## using precomputed fitted values## -------------------------------------------------------------np.random.seed(12345)folds_outer=np.random.choice(a=np.arange(2),size=n,replace=True,p=np.array([0.5,0.5]))## fit the full regressioncv_full.fit(x[folds_outer==1,:],y[folds_outer==1])full_fit=cv_full.best_estimator_.predict(x[folds_outer==1,:])## fit the reduced regressionx_small=np.delete(x[folds_outer==0,:],s,1)# delete the columns in scv_small.fit(x_small,y[folds_outer==0])small_fit=cv_small.best_estimator_.predict(x_small)## get variable importance estimatesnp.random.seed(12345)vimp_precompute=vimpy.vim(y=y,x=x,s=1,f=full_fit,r=small_fit,measure_type="r_squared",folds=folds_outer)## get the point estimate of variable importancevimp_precompute.get_point_est()## get the influence function estimatevimp_precompute.get_influence_function()## get a standard errorvimp_precompute.get_se()## get a confidence intervalvimp_precompute.get_ci()## do a hypothesis test, compute p-valuevimp_precompute.hypothesis_test(alpha=0.05,delta=0)## display the estimates, etc.vimp_precompute.vimp_vimp_precompute.se_vimp_precompute.ci_vimp_precompute.p_value_vimp_precompute.hyp_test_## -------------------------------------------------------------## get variable importance estimates using cross-validation## -------------------------------------------------------------np.random.seed(12345)## set up the vimp objectvimp_cv=vimpy.cv_vim(y=y,x=x,s=1,pred_func=cv_full,V=5,measure_type="r_squared")## get the point estimatevimp_cv.get_point_est()## get the standard errorvimp_cv.get_influence_function()vimp_cv.get_se()## get a confidence intervalvimp_cv.get_ci()## do a hypothesis test, compute p-valuevimp_cv.hypothesis_test(alpha=0.05,delta=0)## display estimates, etc.vimp_cv.vimp_vimp_cv.se_vimp_cv.ci_vimp_cv.p_value_vimp_cv.hyp_test_LogoThe logo was created usinghexSticker,lisa, and apython imagedistributed under the CC0 license. Many thanks to the maintainers of these packages and theColor Lisateam.