Package: timeshap
TimeSHAPTimeSHAP is a model-agnostic, recurrent explainer that builds upon KernelSHAP and
extends it to the sequential domain.
TimeSHAP computes event/timestamp- feature-, and cell-level attributions.
As sequences can be arbitrarily long, TimeSHAP also implements a pruning algorithm
based on Shapley Values, that finds a subset of consecutive, recent events that contribute
the most to the decision.This repository is the code implementation of the TimeSHAP algorithm
present in the paperTimeSHAP: Explaining Recurrent Models through Sequence Perturbationspublished atKDD 2021.Links to the paperhere,
and to the video presentationhere.Install TimeSHAPVia Pippip install timeshapVia GithubClone the repository into a local directory using:git clone https://github.com/feedzai/timeshap.gitMove into the cloned repo and install the package:cd timeshap
pip install .Test your installationStart a Python session in your terminal usingpythonAnd import TimeSHAPimport timeshapTimeSHAP in 30 secondsInputsModel being explained;Instance(s) to explain;Background instance.OutputsLocal pruning output; (explaining a single instance)Local event explanations; (explaining a single instance)Local feature explanations; (explaining a single instance)Global pruning statistics; (explaining multiple instances)Global event explanations; (explaining multiple instances)Global feature explanations; (explaining multiple instances)Model InterfaceIn order for TimeSHAP to explain a model, an entry point must be provided.
ThisCallableentry point must receive a 3-D numpy array,(#sequences; #sequence length; #features)and return a 2-D numpy array(#sequences; 1)with the corresponding score of each sequence.In addition, to make TimeSHAP more optimized, it is possible to return thehidden stateof the model together with the score (if applicable). Although this is optional, we highly recommended it,
as it has a very high impact.
If you choose to return the hidden state, this hidden state should either be:
(seenotebookfor specific examples)a 3-D numpy array,(#rnn layers, #sequences, #hidden_dimension)(classExplainedRNNon notebook);a tuple of numpy arrays that follows the previously described characteristic
(usually used when using stacked RNNs with different hidden dimensions) (classExplainedGRU2Layeron notebook);a tuple of tuples of numpy arrays (usually used when using LSTM's) (classExplainedLSTMon notebook);;
TimeSHAP is able to explain any black-box model as long as it complies with the
previously described interface, including both PyTorch and TensorFlow models,
both examplified in our tutorials (PyTorch,TensorFlow).Example provided in our tutorials:TensorFLowmodel = tf.keras.models.Model(inputs=inputs, outputs=ff2)
f = lambda x: model.predict(x)Pytorch- (Example where model receives and returns hidden states)model_wrapped = TorchModelWrapper(model)
f_hs = lambda x, y=None: model_wrapped.predict_last_hs(x, y)Model WrappersIn order to facilitate the interface between models and TimeSHAP,
TimeSHAP implementsModelWrappers. These wrappers, used on the PyTorchtutorialnotebook, allow for greater flexibility
of explained models as they allow:Batching logic: useful when using very large inputs or NSamples, which cannot fit
on GPU memory, and therefore batching mechanisms are required;Input format/type: useful when your model does not work with numpy arrays. This
is the case of our provided PyToch example;Hidden state logic: useful when the hidden states of your models do not match
the hidden state format required by TimeSHAPTimeSHAP Explanation MethodsTimeSHAP offers several methods to use depending on the desired explanations.
Local methods provide detailed view of a model decision corresponding
to a specific sequence being explained.
Global methods aggregate local explanations of a given dataset
to present a global view of the model.Local ExplanationsPruninglocal_pruning()performs the pruning
algorithm on a given sequence with a given user defined tolerance and returns
the pruning index along the information for plotting.plot_temp_coalition_pruning()plots the pruning
algorithm information calculated bylocal_pruning().Event level explanationslocal_event()calculates event level explanations
of a given sequence with the user-given parameteres and returns the respective
event-level explanations.plot_event_heatmap()plots the event-level explanations
calculated bylocal_event().Feature level explanationslocal_feat()calculates feature level explanations
of a given sequence with the user-given parameteres and returns the respective
feature-level explanations.plot_feat_barplot()plots the feature-level explanations
calculated bylocal_feat().Cell level explanationslocal_cell_level()calculates cell level explanations
of a given sequence with the respective event- and feature-level explanations
and user-given parameteres, returing the respective cell-level explanations.plot_cell_level()plots the feature-level explanations
calculated bylocal_cell_level().Local Reportlocal_report()calculates TimeSHAP
local explanations for a given sequence and plots them.Global ExplanationsGlobal pruning statisticsprune_all()performs the pruning
algorithm on multiple given sequences.pruning_statistics()calculates the pruning
statistics for several user-given pruning tolerances using the pruning
data calculated byprune_all(), returning apandas.DataFramewith the statistics.Global event level explanationsevent_explain_all()calculates TimeSHAP
event level explanations for multiple instances given user defined parameters.plot_global_event()plots the global event-level explanations
calculated byevent_explain_all().Global feature level explanationsfeat_explain_all()calculates TimeSHAP
feature level explanations for multiple instances given user defined parameters.plot_global_feat()plots the global feature-level
explanations calculated byfeat_explain_all().Global reportglobal_report()calculates TimeSHAP
explanations for multiple instances, aggregating the explanations on two plots
and returning them.TutorialIn order to demonstrate TimeSHAP interfaces and methods, you can consultAReM.ipynb.
In this tutorial we get an open-source dataset, process it, train
Pytorch recurrent model with it and use TimeSHAP to explain it, showcasing all
previously described methods.Additionally, we also train a TensorFlow model on the same datasetAReM_TF.ipynb.Repository Structurenotebooks- tutorial notebooks demonstrating the package;src/timeshap- the package source code;src/timeshap/explainer- TimeSHAP methods to produce the explanationssrc/timeshap/explainer/kernel- TimeSHAPKernelsrc/timeshap/plot- TimeSHAP methods to produce explanation plotssrc/timeshap/utils- util methods for TimeSHAP executionsrc/timeshap/wrappers- Wrapper classes for models in order to ease TimeSHAP explanationsCiting TimeSHAP@inproceedings{bento2021timeshap,
    author = {Bento, Jo\~{a}o and Saleiro, Pedro and Cruz, Andr\'{e} F. and Figueiredo, M\'{a}rio A.T. and Bizarro, Pedro},
    title = {TimeSHAP: Explaining Recurrent Models through Sequence Perturbations},
    year = {2021},
    isbn = {9781450383325},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    url = {https://doi.org/10.1145/3447548.3467166},
    doi = {10.1145/3447548.3467166},
    booktitle = {Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining},
    pages = {2565–2573},
    numpages = {9},
    keywords = {SHAP, Shapley values, TimeSHAP, XAI, RNN, explainability},
    location = {Virtual Event, Singapore},
    series = {KDD '21}
}