Package: shap-bootstrap
Shapley BootstrappingFlexibleEffectiveExplainableShapley bootstrapping is a novel machine learning methodology that harmonizes ensemble learners with Shapley values. For detailed explanation, see my [thesis].InstallShapley-bootstrapping can be installed viaPyPipip install shap-bootstrapThis library automatically installs the following dependancies:1.scipy
2.pandas
3.openml
4.xgboost
5.scikit-learn
6.seaborn
7.shap
8.matplotlibOverviewThis library includes the implementations of eight pipelines from [paper]. These pipelines are:Each of these pipelines are already implemented in the library(except 3 and 6, which are special cases) and can be directly used to train and predict over datasets.Usage (Flexible)Name of the module isshap_bootstrap. From this module, you can import the following sub-modules:building_blocksclustercustom_pipelinedatasetsutilsvisualisationIn order to follow the experiments from the [paper] or start with a quick example, you can use thecustom_pipelinemodule for creating a pre-built pipeline using custom building blocks.These 5 building_block classes are:ProcessingBlockThis block is for input pre-processing. Used for input scaling,imputing and train-test splittingExplainerBlockThis block trains the Shap Explainer. Currently trains either Linear Regressor(keywordLinear) or XGBoost Regressor (keywordXGBoost), but will be changed in the future.ClusterBlockThis block takes two algorithms : One unsupervised clustering algorithm and one classifier model. In our research we have used K-Means and K-NN , but these models can be changed to any other for experimentation.EnsembleBlockEnsemble block trains a set of individual XGBoost regressors over the clustered data. Currently either Linear Regressor(keywordLinear) or XGBoost Regressor (keywordXGBoost) but will be made parametric.ReduceBlockReduce block runs PCA over the data to project into lower dimensionality. Currently, PCA is fitted until 95% variance ratio is captured.In an example below, we will implement Branch8 which uses dimensionality reduction, clustering and ensemble training using these building blocks.Workflow of branch8The process sequence of this workflow is as follows:Explainer Block inputs datasets, fits aShapley Explainerand calculates Shapley values of instancesReduce Block projects feature space to lower dimensionality using PCA with 95% explained variance ratioCluster Block exercises given clustering algorithm(K-means in our case) and labels instancesOne-to-one mapper maps these labels back to original instancesEnsemble block trains a model(XGBoost in this case) over each clusterimportmathfromshap_bootstrap.building_blocksimport*fromshap_bootstrapimportdatasetsfromshap_bootstrap.custom_pipelineimportB1_Branch_Pipeline,B8_Branch_Pipelinefromsklearn.clusterimportKMeansfromsklearn.neighborsimportKNeighborsClassifierfromsklearn.model_selectionimporttrain_test_splitfromsklearnimportmetrics# Returns boston dataset & train-test splitX,y,name=datasets.returnDataset(9)X_train,X_test,y_train,y_test=prepare_pipeline_data(X,y,random_state=42)# Building blocks to be used in pipeline# All algorithms can be changed with other models except PCAexplainer_type_b1='XGBoost'explainer_type_b8='Linear'model_type='XGBoost'nClusters=4processing_block_b1=ProcessingBlock()explainer_block_b1=ExplainerBlock(explainer_type_b1)cluster_block_b1=ClusterBlock(nClusters,KMeans(n_clusters=nClusters,random_state=0),KNeighborsClassifier(n_neighbors=nClusters))ensemble_block_b1=EnsembleBlock(model_type)# Instantianate Branch 8 pipelinebranch1=B1_Branch_Pipeline(processing_block_b1,explainer_block_b1,cluster_block_b1,ensemble_block_b1)processing_block_b8=ProcessingBlock()explainer_block_b8=ExplainerBlock(explainer_type_b8)reduce_block_b8=ReduceBlock(PCA(1))cluster_block_b8=ClusterBlock(nClusters,KMeans(n_clusters=nClusters,random_state=0),KNeighborsClassifier(n_neighbors=nClusters))ensemble_block_b8=EnsembleBlock(model_type)# Instantianate Branch 8 pipelinebranch8=B8_Branch_Pipeline(processing_block_b8,explainer_block_b8,reduce_block_b8,cluster_block_b8,ensemble_block_b8)# Fit and predictbranch1.fit(X_train,y_train)y_pred_b1=branch1.predict(X_test)err_b1=math.sqrt(metrics.mean_squared_error(y_test,y_pred_b1))branch8.fit(X_train,y_train)y_pred_b8=branch8.predict(X_test)err_b8=math.sqrt(metrics.mean_squared_error(y_test,y_pred_b8))This code snippet implements branch8, trains  over thestudent_gradesdataset and makes predictions. Now, we can further evaluate our predictions.Evaluation of results (Effective)We can visualise model predictions via:ax=plt.subplot()ax.scatter(x=y_test,y=y_pred_b1)ax.scatter(x=y_test,y=y_pred_b8)ax.plot([0,25],[0,25],color='red',linestyle='--')ax.set_xlabel('True label')ax.set_ylabel('Pipeline predictions')ax.set_title('Divergence of predictions from true label')new_labels=['Identity','XGBoost Model - RMSE:{:.3f}'.format(err_b1),'Shap-bootstrap - RMSE:{:.3f}'.format(err_b8)]ax.legend(new_labels)Plot of prediction error, blue dots is our model, red line is the best predictionIn the plot, we observe that our proposed methodology can improve the performance of an existing XGBoost model, by fitting the data better (Orange points lie closer to the identity line). We improved the RMSE value by 5%, but this is not a major improvement.Interpreting Features (Explainable)We can further check the Shapley values as further interpretation of feature importances:explainer=branch8.explainer_block.explainershap.initjs()shap.summary_plot(explainer.shap_values(X_train),X_train,X.columns,max_display=10)which outputs following plot :Here, we see the features ranked descending by the sum of Shapley value magnitudes over all samples. Namely, they are ranked in feature importance top to bottom. For example, featureG2is positively correlated with output, such that high values ofG2increase the output label and vice versa.With this capability, we can inspect on feature importances, which will aid the interpretability of the model.Model-agnostic functionality (Flexible)The training pipelines take machine learning models as arguments in instantianation. Therefore, it is possible to run the experimentations with different models. Shap-bootstrap offers a flexbility in implementation.