Introducing Transformers Interpret — Explainable AI for Transformers | by Charles Pierse | Towards Data Science

TL:DR: Transformers Interpret brings explainable AI to the transformers package with just 2 lines of code. It allows you to get word attributions and visualizations for those attributions simply. Right now the package supports all transformer models with a sequence classification head.
Check out the project here: https://github.com/cdpierse/transformers-interpret
Model explainability and state-of-the-art research seem to constantly be engaged in a tug of war. Complex models are difficult to interpret and simple models are often much easier to understand.
Explainable AI is one of the most vital emerging subfields in machine learning today. For both ethical, practical, and - in the corporate world — legal reasons we must be able to explain the inner workings of so called “black box” models, it’s vital to the long term success of AI’s adoption into our lives.
Research in explainable AI is ongoing and there are a number of fantastic projects and groups out there doing work. The library on which transformers interpret is built is called Captum which is a package designed for model interpretability in pytorch. What I love about Captum is that it effectively combines all the leading research in explainable AI into a single package, and of course it is optimized for pytorch which is a big plus. While transformers interpret is focused solely on NLP, Captum is multi modal and works with text, computer vision, and tabular data. The team at Captum have also done a fantastic job giving write ups of all the algorithms they are using throughout the package, it’s certainly worth checking out if you are interested in understanding the internals of transformers interpret.
Much like the design philosophy behind Hugging Faces Transformers package transformers interpret was designed with ease of use at the forefront. It is opinionated in its selection of attribution methods and how it summarizes attributions. All of this allows end users to get word attributions and visualizations for their model’s output in just 2 lines of code. Right now the package has support for all models with sequence classification heads — so (should) work for all classification models. Support for question answering models and NER models is planned.
Let’s jump into a hands on example. To install the package:
pip install transformers-interpret
With the package installed we are going to start by instantiating a transformers model and tokenizer. I’ve chosen to go with distilbert-base-uncased-finetuned-sst-2-english it’s a distilbert model finetuned on a sentiment analysis task, I chose this model mostly because it is popular and lightweight compared to some of the other behemoths out there.
from transformers import AutoModelForSequenceClassification, AutoTokenizermodel_name = "distilbert-base-uncased-finetuned-sst-2-english"model = AutoModelForSequenceClassification.from_pretrained(model_name)tokenizer = AutoTokenizer.from_pretrained(model_name)
With both the model and tokenizer in hand we can now go about creating an explainer, and it’s pretty simple:
from transformers_interpret import SequenceClassificationExplainercls_explainer = SequenceClassificationExplainer("I love you, I like you", model, tokenizer)
That’s all it takes. To get the attributions for the sentence “I love you, I like you” we simply call:
attributions = cls_explainer()
If we want to know which class was predicted:
>>> cls_explainer.predicted_class_name'POSITIVE'
To see the raw numeric attributions:
>>> attributions.word_attributions[('BOS_TOKEN', 0.0),('I', 0.46820529249283205),('love', 0.46061853275727177),('you', 0.566412765400519),(',', -0.017154456486408547),('I', -0.053763869433472),('like', 0.10987746237531228),('you', 0.48221682341218103),('EOS_TOKEN', 0.0)]
This is interesting we can see that the model is placing a significant amount of attention on “I love you” while “I like you” is not as important. Given how self-attention works this seems to fit with the intuition that the word “love” is increasing the importance of the words surrounding it’s context.
The numeric attributions can be difficult to read especially for much longer sentences. The package also has an inbuilt visualize method built on top of Captums’ visualization to give easy to interpret visual explanations of word attributions.
If you are working in a jupyter notebook calling the method will display the visualization in-line, if you are running from a script simply pass a html filename and you can view the output file in your browser.
cls_explainer.visualize("distilbert_example.html")
This will produce an output that looks like this:
If you want to generate further attributions there is no need to create another explainer, you can simply pass some new text to the existing explainer and new attributions will be created
attributions = cls_explainer("I hate you, I dislike you")
You are not limited to generating attributions for the predicted class, this is the default behavior but can be overridden by passing either a class name or index of the class you would like attributions for:
attributions = cls_explainer(class_name="NEGATIVE")
This is particularly useful for multiclass classification models.
For a more detailed example of this check out this multiclass classification notebook
Below is a full working example of the above code.
At the time of writing the package only supports classification models. However question answering models are also very possible to implement as are NER models and both of these are currently planned features.
I’m also interested in exploring the feasibility of explanations for causal language models and multiple choice models.
Captum has a variety of attributions methods, the default in transformers interpret is Layer Integrated Gradients, with the attribution scores effectively being a summary or average of each layers’ attributions. Captum recently added support for performing Layer Integrated Gradients on specific layer(s) of the transformer, so I am hoping to add the option for users to get attributions on a layer by layer basis which would allow even deeper inspection of a models’ internals.
Check out the repo for any updates regarding future releases.
If you have any questions, suggestions, or would like to make a contribution (please do 😁) please get in touch.
I’d also highly suggest checking out Captum if you find model explainability and interpretability interesting. They are doing amazing and important work.
Thanks for reading!