Package: inseq
Intepretability for Sequence Generation Models 🔍Paper|Documentation|Tutorial|PyPI PackageInseq is a Pytorch-based hackable toolkit to democratize the access to common post-hocinterpretability analyses ofsequence generation models.InstallationInseq is available on PyPI and can be installed withpipfor Python >= 3.9, <= 3.11:# Install latest stable versionpipinstallinseq# Alternatively, install latest development versionpipinstallgit+https://github.com/inseq-team/inseq.gitInstall extras for visualization in Jupyter Notebooks and 🤗 datasets attribution aspip install inseq[notebook,datasets].Dev InstallationTo install the package, clone the repository and run the following commands:cdinseq
makepoetry-download# Download and install the Poetry package managermakeinstall# Installs the package and all dependenciesIf you have a GPU available, usemake install-gputo install the latesttorchversion with GPU support.For library developers, you can use themake install-devcommand to install and its GPU-friendly counterpartmake install-dev-gputo install all development dependencies (quality, docs, extras).After installation, you should be able to runmake fast-testandmake lintwithout errors.FAQ InstallationInstalling thetokenizerspackage requires a Rust compiler installation. You can install Rust fromhttps://rustup.rsand add$HOME/.cargo/envto your PATH.Installingsentencepiecerequires various packages, install withsudo apt-get install cmake build-essential pkg-configorbrew install cmake gperftools pkg-config.Inseq does not work with older versions ofjaxlib.
Install a compatible version withpoetry install --with jax.Example usage in PythonThis example uses the Integrated Gradients attribution method to attribute the English-French translation of a sentence taken from the WinoMT corpus:importinseqmodel=inseq.load_model("Helsinki-NLP/opus-mt-en-fr","integrated_gradients")out=model.attribute("The developer argued with the designer because her idea cannot be implemented.",n_steps=100)out.show()This produces a visualization of the attribution scores for each token in the input sentence (token-level aggregation is handled automatically). Here is what the visualization looks like inside a Jupyter Notebook:Inseq also supports decoder-only models such asGPT-2, enabling usage of a variety of attribution methods and customizable settings directly from the console:importinseqmodel=inseq.load_model("gpt2","integrated_gradients")model.attribute("Hello ladies and",generation_args={"max_new_tokens":9},n_steps=500,internal_batch_size=50).show()Features🚀 Feature attribution of sequence generation for mostForConditionalGeneration(encoder-decoder) andForCausalLM(decoder-only) models from 🤗 Transformers🚀 Support for multiple feature attribution methods, sourced in part fromCaptum🚀 Post-processing of attribution maps viaAggregatorclasses.🚀 Attribution visualization in notebooks, browser and command line.🚀 Attribute single examples or entire 🤗 datasets with the Inseq CLI.🚀 Custom attribution of target functions, supporting advanced use cases such as contrastive and uncertainty-weighted feature attributions.🚀 Extraction and visualization of custom step scores (e.g. probability, entropy) alongsides attribution maps.Supported methodsUse theinseq.list_feature_attribution_methodsfunction to list all available method identifiers andinseq.list_step_functionsto list all available step functions. The following methods are currently supported:Gradient-based attributionsaliency:Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps(Simonyan et al., 2013)input_x_gradient:Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps(Simonyan et al., 2013)integrated_gradients:Axiomatic Attribution for Deep Networks(Sundararajan et al., 2017)deeplift:Learning Important Features Through Propagating Activation Differences(Shrikumar et al., 2017)gradient_shap:A unified approach to interpreting model predictions(Lundberg and Lee, 2017)discretized_integrated_gradients:Discretized Integrated Gradients for Explaining Language Models(Sanyal and Ren, 2021)sequential_integrated_gradients:Sequential Integrated Gradients: a simple but effective method for explaining language models(Enguehard, 2023)Internals-based attributionattention: Attention Weight Attribution, fromNeural Machine Translation by Jointly Learning to Align and Translate(Bahdanau et al., 2014)Perturbation-based attributionocclusion:Visualizing and Understanding Convolutional Networks(Zeiler and Fergus, 2014)lime:"Why Should I Trust You?": Explaining the Predictions of Any Classifier(Ribeiro et al., 2016)Step functionsStep functions are used to extract custom scores from the model at each step of the attribution process with thestep_scoresargument inmodel.attribute. They can also be used as targets for attribution methods relying on model outputs (e.g. gradient-based methods) by passing them as theattributed_fnargument. The following step functions are currently supported:logits: Logits of the target token.probability: Probability of the target token. Can also be used for log-probability by passinglogprob=True.entropy: Entropy of the predictive distribution.crossentropy: Cross-entropy loss between target token and predicted distribution.perplexity: Perplexity of the target token.contrast_logits/contrast_prob: Logits/probabilities of the target token when different contrastive inputs are provided to the model. Equivalent tologits/probabilitywhen no contrastive inputs are provided.contrast_logits_diff/contrast_prob_diff: Difference in logits/probability between original and foil target tokens pair, can be used for contrastive evaluation as incontrastive attribution(Yin and Neubig, 2022).pcxmi: Point-wise Contextual Cross-Mutual Information (P-CXMI) for the target token given original and contrastive contexts(Yin et al. 2021).kl_divergence: KL divergence of the predictive distribution given original and contrastive contexts. Can be restricted to most likely target token options using thetop_kandtop_pparameters.in_context_pvi: In-context Pointwise V-usable Information (PVI) to measure the amount of contextual information used in model predictions(Lu et al. 2023).mc_dropout_prob_avg: Average probability of the target token across multiple samples usingMC Dropout(Gal and Ghahramani, 2016).top_p_size: The number of tokens with cumulative probability greater thantop_pin the predictive distribution of the model.The following example computes contrastive attributions using thecontrast_prob_diffstep function:importinseqattribution_model=inseq.load_model("gpt2","input_x_gradient")# Perform the contrastive attribution:# Regular (forced) target -> "The manager went home because he was sick"# Contrastive target      -> "The manager went home because she was sick"out=attribution_model.attribute("The manager went home because","The manager went home because he was sick",attributed_fn="contrast_prob_diff",contrast_targets="The manager went home because she was sick",# We also visualize the corresponding step scorestep_scores=["contrast_prob_diff"])out.show()Refer to thedocumentationfor an example including custom function registration.Using the Inseq clientThe Inseq library also provides useful client commands to enable repeated attribution of individual examples and even entire 🤗 datasets directly from the console. See the available options by typinginseq -hin the terminal after installing the package.For now, two commands are supported:ìnseq attribute: Wraps theattributemethod shown above, requires explicit inputs to be attributed.inseq attribute-dataset: Enables attribution for a full dataset using Hugging Facedatasets.load_dataset.Both commands support the full range of parameters available forattribute, attribution visualization in the console and saving outputs to disk.Example:The following command can be used to perform attribution (both source and target-side) of Italian translations for a dummy sample of 20 English sentences taken from the FLORES-101 parallel corpus, using a MarianNMT translation model from Hugging Facetransformers. We save the visualizations in HTML format in the fileattributions.html. See the--helpflag for more options.inseqattribute-dataset\--model_name_or_pathHelsinki-NLP/opus-mt-en-it\--attribution_methodsaliency\--do_prefix_attribution\--dataset_nameinseq/dummy_enit\--input_text_fielden\--dataset_split"train[:20]"\--viz_pathattributions.html\--batch_size8\--hidePlanned Development⚙️ Support more attention-based and occlusion-based feature attribution methods (documented in#107and#108).⚙️ Interoperability withferretfor attribution plausibility and faithfulness evaluation.⚙️ Rich and interactive visualizations in a tabbed interface usingGradio Blocks.ContributingOur vision for Inseq is to create a centralized, comprehensive and robust set of tools to enable fair and reproducible comparisons in the study of sequence generation models. To achieve this goal, contributions from researchers and developers interested in these topics are more than welcome. Please see ourcontributing guidelinesand ourcode of conductfor more information.Citing InseqIf you use Inseq in your research we suggest to include a mention to the specific release (e.g. v0.4.0) and we kindly ask you to cite our reference paper as:@inproceedings{sarti-etal-2023-inseq,title="Inseq: An Interpretability Toolkit for Sequence Generation Models",author="Sarti, Gabriele  andFeldhus, Nils  andSickert, Ludwig  andvan der Wal, Oskar andNissim, Malvina andBisazza, Arianna",booktitle="Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)",month=jul,year="2023",address="Toronto, Canada",publisher="Association for Computational Linguistics",url="https://aclanthology.org/2023.acl-demo.40",doi="10.18653/v1/2023.acl-demo.40",pages="421--435",}Research using InseqInseq has been used in various research projects. A list of known publications that use Inseq to conduct interpretability analyses of generative models is shown below. If you know more, please let us know or submit a pull request (last updated: December 2023).2023Inseq: An Interpretability Toolkit for Sequence Generation Models(Sarti et al., 2023)Are Character-level Translations Worth the Wait? Comparing Character- and Subword-level Models for Machine Translation(Edman et al., 2023)Response Generation in Longitudinal Dialogues: Which Knowledge Representation Helps?(Mousavi et al., 2023)Quantifying the Plausibility of Context Reliance in Neural Machine Translation(Sarti et al., 2023)A Tale of Pronouns: Interpretability Informs Gender Bias Mitigation for Fairer Instruction-Tuned Machine Translation(Attanasio et al., 2023)Assessing the Reliability of Large Language Model Knowledge(Wang et al., 2023)Attribution and Alignment: Effects of Local Context Repetition on Utterance Production and Comprehension in Dialogue(Molnar et al., 2023)