Package: transparency
The "Transparency" LibraryScalable and Fast, local (single level) and global (population level) prediction explanation of:Ensemble trees (e.g., XGB, GBM, RF, and Decision tree)Generalized linear models GLM (support for various families, link powers, and variance powers, e.g., logistic regression)implemented for models in:Python (Scikit-Learn)Pyspark (Scala and Pyspark)Installation:pip install transparencyadditional step for Spark users:Add this jar to spark classpath :https://github.com/alvinhenrick/spark_model_explainer/releases/download/v.0.0.1/spark_model_explainer-assembly-0.0.1.jar(Maven repository release will soon be supported :https://github.com/clarifyhealth/spark_model_explainer)Transformer Set- Scikit-Learn Ensemble Tree Explainer Transformerfromtransparency.python.explainer.ensemble_treeimportEnsembleTreeExplainerTransformerexpl=EnsembleTreeExplainerTransformer(estimator)X_test_df=expl.transform(X_test_df)estimator: the ensemble tree estimator that has been trained (e.g., random forest, gbm, or xgb)X_test: a Pandas dataframe with features as columns and samples as rows
The resulting X_test_df will have 3 added columns: 'prediction', 'feature_contributions' and 'intercept_contribution':'feature_contributions': column of nested arrays with feature contributions (1 array per row)'intercept_contribution': column of the same scaler value representing the contribution of the intercept
sum(contributions) + contrib_intercept for each row equals the prediction for that row.- Scikit-Learn Generalized Linear Model (e.g., Logistic regression) Explainer Transformerfromtransparency.python.explainer.glmimportGLMExplainerTransformerexpl=GLMExplainerTransformer(estimator)X_test_df=expl.transform(X_test_df,output_proba=False)estimator: the glm estimator that has been trained (e.g., logistic regression)X_test: a Pandas dataframe with features as columns and samples as rows
The resulting X_test_df will have 3 added columns: 'prediction', 'feature_contributions' and 'intercept_contribution':'feature_contributions': column of nested arrays with feature contributions (1 array per row)'intercept_contribution': column of the same scaler value representing the contribution of the intercept
sum(contributions) + contrib_intercept for each row equals the prediction for that row.if output_proba is set to True, for the case of logistic regression, the output prediction and its corresponding explanation will be proba instead of the classification result- Pyspark Ensemble Tree Explainer Transformerfromtransparency.spark.prediction.explainer.treeimportEnsembleTreeExplainTransformerEnsembleTreeExplainTransformer(predictionView=predictions_view,featureImportanceView=features_importance_view,modelPath=rf_model_path,label=label_column,dropPathColumn=True,isClassification=classification,ensembleType=ensemble_type)Path to load modelmodelPathSupportedensembleTypedctgbtrfxgboost4jThe feature importance extracted from Apache Spark Model Meta Data.featureImportanceViewReference this python script :testutil.common.get_feature_importanceFeature_IndexFeatureOriginal_FeatureImportanceThe transformer append 3 main column to the prediction viewcontrib_column ==>f"{prediction_{label_column}_contrib:array of contributionscontrib_column_sum ==>f"{contrib_column}_sum"contrib_column_intercept ==>f"{contrib_column}_intercept"- Pyspark Generalized Linear Model (GLM) Explainer Transformerfromtransparency.spark.prediction.explainer.treeimportGLMExplainTransformerGLMExplainTransformer(predictionView=predictions_view,coefficientView=coefficients_view,linkFunctionType=link_function_type,label=label_column,nested=True,calculateSum=True,family=family,variancePower=variance_power,linkPower=link_power)SupportedlinkFunctionTypelogLinkpowerHalfLinkidentityLinklogitLinkinverseLinkotherPowerLinkThe feature coefficient extracted from Apache Spark Model Meta Data.coefficientViewReference this python script :testutil.common.get_feature_coefficientsFeature_IndexFeatureOriginal_FeatureCoefficientThe transformer append 3 main column to the prediction viewcontrib_column ==>f"{prediction_{label_column}_contrib:array of contributionscontrib_column_sum ==>f"{contrib_column}_sum"contrib_column_intercept ==>f"{contrib_column}_intercept"Example NotebooksPython (Scikit-Learn) Ensemble Tree Explain Example:https://github.com/imanbio/transparency/blob/master/examples/notebooks/python/python_ensemble_tree_explainer_samples.ipynbPython (Scikit-Learn) Generalized Linear Model Explain Example:https://github.com/imanbio/transparency/blob/master/examples/notebooks/python/python_glm_explainer_samples.ipynbPySpark GLM Explain Example:https://github.com/imanbio/transparency/blob/master/examples/notebooks/spark/pyspark_glm_explain.ipynbPySpark Random Forest Explain Example:https://github.com/imanbio/transparency/blob/master/examples/notebooks/spark/pyspark_random_forest_explain.ipynbAuthorsIman Hajihttps://www.linkedin.com/in/imanhajiImran Qureshihttps://www.linkedin.com/in/imranq2/Alvin Henrickhttps://www.linkedin.com/in/alvinhenrick/LicenseApache License Version 2.0