Repository: shap/shap

Issue #3036: [Meta-issue] Notebooks are outdated / non-runnable
Created: 2023-06-23T18:41:48Z
Background
We should also make sure that our documentation is kept up to date.
A scour through the open issues in this repo and also on StackOverflow shows that the outdated documentation (or lackthereof) is causing confusion among our users.
Just some examples:

#2786
https://stackoverflow.com/q/76522186


Plan
The plan here is to thoroughly go through each and every notebook example that we have, to:

Run the notebook from top-to-bottom and ensure there are no errors.
Update the prose where necessary to provide better clarity and fix any typos
Update the code (where necessary and appropriate) to demonstrate up-to-date API & python features, such as:

Replacing the deprecated boston dataset
Using the Explanation API for plotting (e.g. shap.plots.beeswarm rather than shap.summary_plot)



An example of such a PR: #3037
Useful info for contributors
To potential contributors (thank you in advance!), please stick to one notebook per PR when contributing.
Also, any help in updating the GPU-related notebooks would be very much appreciated.
We have two linting checks for notebooks, each of which has an "exclude list" that ignores notebooks that haven't been cleaned. When a notebook has been fixed up, it should be removed from the "exclude" list in these places:

The run_notebooks_timeout job in scripts/run_notebooks_timeouts.py
The nbcheckorder job in .pre-commit-config.yaml

For more details on how to preview the built documentation, see the contributing guide.
TODO

notebooks

text_examples

language_modelling

 Language Modeling Explanation Demo.ipynb (#3691)


text_generation

 Open Ended GPT2 Text Generation Explanations.ipynb


question_answering

 Explaining a Question Answering Transformers Model.ipynb


translation

 Machine Translation Explanations.ipynb


summarization

 Abstractive Summarization Explanation Demo.ipynb


text_entailment

 Textual Entailment Explanation Demo.ipynb


sentiment_analysis

 Using custom functions and tokenizers.ipynb
 Emotion classification multiclass example.ipynb
 Positive vs. Negative Sentiment Classification.ipynb
 Keras LSTM for IMDB Sentiment Classification.ipynb




image_examples

image_captioning

 Image Captioning using Azure Cognitive Services.ipynb
 Image Captioning using Open Source.ipynb


image_classification

 Explain ResNet50 using the Partition explainer.ipynb
 PyTorch Deep Explainer MNIST example.ipynb (#3591)
 Explain MobilenetV2 using the Partition explainer (PyTorch).ipynb
 Multi-class ResNet50 on ImageNet (TensorFlow)-checkpoint.ipynb
 Front Page DeepExplainer MNIST Example.ipynb (#3393)
 Multi-class ResNet50 on ImageNet (TensorFlow).ipynb
 Multi-input Gradient Explainer MNIST Example.ipynb
 Explain an Intermediate Layer of VGG16 on ImageNet (PyTorch).ipynb
 Image Multi Class.ipynb
 Explain an Intermediate Layer of VGG16 on ImageNet.ipynb




benchmarks

tabular

 Tabular Prediction Benchmark Demo.ipynb (#3338)
 Benchmark XGBoost explanations.ipynb (#3339)


others

 Benchmark Debug Mode.ipynb


image

 Image Multiclass Classification Benchmark Demo.ipynb


text

 Machine Translation Benchmark Demo.ipynb
 Abstractive Summarization Benchmark Demo.ipynb
 Text Emotion Multiclass Classification Benchmark Demo.ipynb




overviews

 Explaining quantitative measures of fairness.ipynb
 Be careful when interpreting predictive models in search of causal insights.ipynb
 An introduction to explainable AI with Shapley values.ipynb


genomic_examples

 DeepExplainer Genomics Example.ipynb (#3458)


api_examples

maskers

 custom.ipynb


explainers

 Permutation.ipynb
 Exact.ipynb
 GPUTree.ipynb


plots

 text.ipynb
 decision_plot.ipynb
 scatter.ipynb #3752
 waterfall.ipynb
 beeswarm.ipynb
 violin.ipynb
 image.ipynb
 bar.ipynb  #3523
 heatmap.ipynb




tabular_examples

model_agnostic

 Squashing Effect.ipynb
 Census income classification with scikit-learn.ipynb
 Simple Kernel SHAP.ipynb
 Multioutput Regression SHAP.ipynb
 Iris classification with scikit-learn.ipynb
 Diabetes regression.ipynb
 Simple Boston California Demo.ipynb (#3332)


tree_based_models

 League of Legends Win Prediction with XGBoost.ipynb (#3275)
 Understanding Tree SHAP for Simple Models.ipynb (#3278, #3749)
 Census income classification with XGBoost.ipynb
 Example of loading a custom tree model into SHAP.ipynb (#3304)
 Census income classification with LightGBM.ipynb (#3303)
 Perfomance Comparison.ipynb
 NHANES I Survival Model.ipynb (#3395)
 Python Version of Tree SHAP.ipynb (#3335)
 Catboost tutorial.ipynb (#3214)
 Scatter Density vs. Violin Plot Comparison.ipynb (#3396)
 Explaining a simple OR function.ipynb (#3037)
 Basic SHAP Interaction Value Example in XGBoost.ipynb (#3346)
 Force Plot Colors.ipynb (#3336)
 Fitting a Linear Simulation with XGBoost.ipynb
 Front page example (XGBoost).ipynb (#3337)
 Explaining the Loss of a Model.ipynb


neural_networks

 Census income classification with Keras.ipynb


linear_models

 Sentiment Analysis with Logistic Regression.ipynb (#3127)
 Explaining a model that uses standardized features.ipynb (#3112)
 Math behind LinearExplainer with correlation feature perturbation.ipynb

Duplicate issue #2880:   AttributeError: module 'shap.models' has no attribute 'TeacherForcingLogits'
Created: 2023-03-18T16:08:02Z
Hello,
I am using Shap 0.41.0, the latest at the time of writing this. I think there needs to be a code update to run_masker so that instead of calling:
wrapped_model = shap.models.TeacherForcingLogits(f, similarity_model=model, similarity_tokenizer=tokenizer)
The team might want to change it to:
wrapped_model = shap.models.TeacherForcing.get_teacher_forced_logits(f, similarity_model=model, similarity_tokenizer=tokenizer)
Here is the error I was getting:
AttributeError                            Traceback (most recent call last)
Cell In[10], line 2
1 # SHAP explanation using masking option "blur(pixel_size, pixel_size)" for blurring
----> 2 shap_values = run_masker(X, mask_value="blur(56,56)")
Cell In[9], line 50, in run_masker(X, mask_value, max_evals, batch_size, fixed_context)
47 masker = shap.maskers.Image(mask_value, X[index].shape)
49 # wrap model with TeacherForcingLogits class
---> 50 wrapped_model = shap.models.TeacherForcingLogits(f, similarity_model=model, similarity_tokenizer=tokenizer)
52 # build a partition explainer with wrapped_model and image masker
53 explainer = shap.Explainer(wrapped_model, masker)
AttributeError: module 'shap.models' has no attribute 'TeacherForcingLogits'

Closing comment (2023-07-08T10:02:21Z): @YuanningEric Not yet. But from what I can see, it's related to #3036 . It's not very clear in the issue report, but I'm assuming the problem here pertains to the Image Captioning notebooks found here: https://shap.readthedocs.io/en/latest/example_notebooks/image_examples/image_captioning/Image%20Captioning%20using%20Open%20Source.html, and not any code in the codebase.
Does @dancbeaulieu 's suggestion apply for you? It's a notebook, so you can just change it and try it yourself.
If you manage to get the notebook working, we would really appreciate if you can submit a PR in the spirit of #3036.