Package: path-explain
Path ExplainA repository for explaining feature importances and feature interactions in deep neural networks using path attribution methods.This repository contains tools to interpret and explain machine learning models usingIntegrated GradientsandExpected Gradients. In addition, it contains code to explaininteractionsin deep networks using Integrated Hessians and Expected Hessians - methods that we introduced in our most recent paper:"Explaining Explanations: Axiomatic Feature Interactions for Deep Networks". If you use our work to explain your networks, please cite this paper.@article{janizek2020explaining,
  author  = {Joseph D. Janizek and Pascal Sturmfels and Su-In Lee},
  title   = {Explaining Explanations: Axiomatic Feature Interactions for Deep Networks},
  journal = {Journal of Machine Learning Research},
  year    = {2021},
  volume  = {22},
  number  = {104},
  pages   = {1-54},
  url     = {http://jmlr.org/papers/v22/20-1223.html}
}This repository contains two important directories: thepath_explaindirectory, which contains the packages used to interpret and explain machine learning models, and theexamplesdirectory, which contains many examples using thepath_explainmodule to explain different models on different data types.InstallationThe easiest way to install this package is by using pip:pip install path-explainAlternatively, you can clone this repository to re-run and explore the examples provided.CompatibilityThis package was written to support TensorFlow 2.0 (in eager execution mode) with Python 3. We have no current plans to support earlier versions of TensorFlow or Python.APIAlthough we don't yet have formal API documentation, the underlying code does a pretty good job at explaining the API. See the code for generatingattributionsandinteractionsto better understand what the arguments to these functions mean.ExamplesFor a simple, quick example to get started using this repository, see theexample_usage.ipynbnotebook in the top-level directory of this repository. It gives an overview of the functionality provided by this repository. For more advanced examples, keep reading on.Tabular Data using Expected Gradients and Expected HessiansOur repository can easily be adapted to explain attributions and interactions learned on tabular data.# other import statements...frompath_explainimportPathExplainerTF,scatter_plot,summary_plot### Code to train a model would go herex_train,y_train,x_test,y_test=datset()model=...model.fit(x_train,y_train,...)###### Generating attributions using expected gradientsexplainer=PathExplainerTF(model)attributions=explainer.attributions(inputs=x_test,baseline=x_train,batch_size=100,num_samples=200,use_expectation=True,output_indices=0)###### Generating interactions using expected hessiansinteractions=explainer.interactions(inputs=x_test,baseline=x_train,batch_size=100,num_samples=200,use_expectation=True,output_indices=0)###Once we've generated attributions and interactions, we can use the provided plotting modules to help visualize them. First we plot a summary of the top features and their attribution values:### First we need a list of strings denoting the name of each featurefeature_names=...###summary_plot(attributions=attributions,feature_values=x_test,feature_names=feature_names,plot_top_k=10)Second, we plot an interaction our model has learned between maximum achieved heart rate and gender:scatter_plot(attributions=attributions,feature_values=x_test,feature_index='max. achieved heart rate',interactions=interactions,color_by='is male',feature_names=feature_names,scale_y_ind=True)The model used to generate the above interactions is a two layer neural network trained on theUCI Heart Disease Dataset. Interactions learned by this model were featured in our paper. To learn more about this particular model and the experimental setup, seethe notebook used to train and explain the model.Explaining an NLP model using Integrated Gradients and Integrated HessiansAs discussed in our paper, we can use Integrated Hessians to get interactions in language models. We explain a transformer from theHuggingFace Transformers Repository.fromtransformersimportDistilBertTokenizer,TFDistilBertForSequenceClassification,\DistilBertConfig,glue_convert_examples_to_features,\glue_processors# This is a custom explainer to explain huggingface modelsfrompath_explainimportEmbeddingExplainerTF,text_plot,matrix_interaction_plot,bar_interaction_plottokenizer=DistilBertTokenizer.from_pretrained('distilbert-base-uncased')config=DistilBertConfig.from_pretrained('distilbert-base-uncased',num_labels=num_labels)model=TFDistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased',config=config)### Some custom code to fine-tune the model on a sentiment analysis task...max_length=128data,info=tensorflow_datasets.load('glue/sst-2',with_info=True)train_dataset=glue_convert_examples_to_features(data['train'],tokenizer,max_length,'sst-2)valid_dataset=glue_convert_examples_to_features(data['validation'],tokenizer,max_length,'sst-2')...### we won't include the whole fine-tuning code. See the HuggingFace repository for more.### Here we define functions that represent two pieces of the model:### embedding and predictiondefembedding_model(batch_ids):batch_embedding=model.distilbert.embeddings(batch_ids)returnbatch_embeddingdefprediction_model(batch_embedding):# Note: this isn't exactly the right way to use the attention mask.# It should actually indicate which words are real words. This# makes the coding easier however, and the output is fairly similar,# so it suffices for this tutorial.attention_mask=tf.ones(batch_embedding.shape[:2])attention_mask=tf.cast(attention_mask,dtype=tf.float32)head_mask=[None]*model.distilbert.num_hidden_layerstransformer_output=model.distilbert.transformer([batch_embedding,attention_mask,head_mask],training=False)[0]pooled_output=transformer_output[:,0]pooled_output=model.pre_classifier(pooled_output)logits=model.classifier(pooled_output)returnlogits###### We need some data to explainforbatchinvalid_dataset.take(1):batch_input=batch[0]batch_ids=batch_input['input_ids']batch_embedding=embedding_model(batch_ids)baseline_ids=np.zeros((1,128),dtype=np.int64)baseline_embedding=embedding_model(baseline_ids)###### We are finally ready to explain our modelexplainer=EmbeddingExplainerTF(prediction_model)attributions=explainer.attributions(inputs=batch_embedding,baseline=baseline_embedding,batch_size=32,num_samples=256,use_expectation=False,output_indices=1)###### For interactions, the hessian is rather large so we use a very small batch sizeinteractions=explainer.interactions(inputs=batch_embedding,baseline=baseline_embedding,batch_size=1,num_samples=256,use_expectation=False,output_indices=1)###We can plot the learned attributions and interactions as follows. First we plot the attributions:### First we need to decode the tokens from the batch ids.batch_sentences=...### Doing so will depend on how you tokenized your model!text_plot(batch_sentences[0],attributions[0],include_legend=True)Then we plot the interactions:bar_interaction_plot(interactions[0],batch_sentences[0],top_k=5)If you would rather plot the full matrix of attributions rather than the top interactions in a bar plot, our package also supports this. First we show the attributions:text_plot(batch_sentences[1],attributions[1],include_legend=True)And then we show the full interaction matrix. Here we've zeroed out the diagonals so you can better see the off-diagonal terms.matrix_interaction_plot(interaction_list[1],token_list[1])This example - interpretingDistilBERT- was also featured in our paper. You can examine the setup morehere. For more examples, see theexamplesdirectory in this repository.