Sequential Integrated Gradients: a simple but effective method for explaining language models
Year: 2023

Abstract:
Several explanation methods such as Integrated Gradients (IG) can be characterised as pathbased methods, as they rely on a straight line between the data and an uninformative baseline. However, when applied to language models, these methods produce a path for each word of a sentence simultaneously, which could lead to creating sentences from interpolated words either having no clear meaning, or having a significantly different meaning compared to the original sentence. In order to keep the meaning of these sentences as close as possible to the original one, we propose Sequential Integrated Gradients (SIG), which computes the importance of each word in a sentence by keeping fixed every other words, only creating interpolations between the baseline and the word of interest. Moreover, inspired by the training procedure of several language models, we also propose to replace the baseline token "pad" with the trained token "mask". While being a simple improvement over the original IG method, we show on various models and datasets that SIG proves to be a very effective method for explaining language models.1

heading: 1 Introduction
text: Language models such as BERT (Devlin et al., 2018) have demonstrated to be effective on various tasks, for instance on sentiment analysis (Hoang et al., 2019), machine translation (Zhu et al., 2020), text summarization (Liu, 2019) or intent classification (Chen et al., 2019). However, with the increased performance and usage of such models, there has been a parallel drive to develop methods to explain predictions made by these models. Indeed, BERT and its variations are complex models which do not allow a user to easily understand why a certain prediction has been produced. On the other hand, it is important to be able to explain a
1An implementation of this work can be found at https: //github.com/josephenguehard/time_interpret
model’s predictions, especially when this model is used to make high-stake decisions, or when there is a risk of a discriminating bias, for instance when detecting hate speech on social media (Sap et al., 2019).
As a result, developing effective methods to explain not only language models, but also machine learning models in general, has recently gained significant attention. Many different methods have therefore been proposed such as: LIME (Ribeiro et al., 2016), Grad*Inp (Shrikumar et al., 2016), Integrated Gradients (IG) (Sundararajan et al., 2017), DeepLift (Shrikumar et al., 2017) or GradientShap (Lundberg and Lee, 2017). Among these methods, some can be characterised as path-based, which
7555
means that they rely on a straight line between the data and an uninformative baseline. For instance, IG computes gradients on interpolated points along such a path, while DeepLift and GradientShap can be seen as approximations of IG (Ancona et al., 2017; Lundberg and Lee, 2017).
While these methods aim to be used on any type of models and data, some have been tailored to the specificity of language models. For instance, Sanyal and Ren (2021) challenge the use of continuous paths on a word embedding space which is inherently discrete. They propose as a result Discretized Integrated Gradient (DIG), which replaces the continuous straight path with a discretized one, where interpolated points are words.
In our work, we suggest another potential issue when applying path-based explanation methods on language models. These models are usually designed to be used on individual or multiple sentences, in order to perform for instance sentiment analysis or question answering. However, a path-based method applied on such models creates straight lines between each word and a baseline simultaneously. When interpolated points are grouped together to form a sentence, this sentence could have a very different meaning compared with the original one.
As a result, we propose a simple method to alleviate this potential issue: computing the importance of each word in a sentence or a text by keeping fixed every other word and only creating interpolations between the baseline and the word of interest. After computing the importance of each word in this way, we normalise these attributions across the sentence or text we aim to explain. We call this method Sequential Integrated Gradients (SIG), as, although we focus in this work on language models, such a method could be used on any sequential modelling. We also propose to use the token "mask" as a baseline, when possible, as its embedding has been trained to replace part of sentences when training language models. As a result, our method follows closely the training procedure of these models.

heading: 2 Method
text: SIG formulation Let’s define a language model as a function F(x) : Rm×n → R. The input x is here modelled as a sequence of m words, each having n features. These features are usually constructed by an embedding layer. We denote xi the
ith word of a sentence (or of a text, depending on the input of the model), and xij the jth feature of the ith word. The output of F is a value in R, which is, in our experiments, a measure of the sentiment for a given sentence. We now define the baseline for each word xi as xi = (x1, ..., <mask>, ..., xm). The baseline is therefore identical to x except at the ith position, where the word xi is replaced by the embedding of the word "mask"2, a token used in many language model to replace part of the sentence during training. Moreover, we use the notation xi instead of xi as xi corresponds to an entire sentence, not to be mistaken with a single word like xi.
In this setting, we keep the baseline as similar to the original sentence as possible, only changing the word of interest. This method of explaining a word is also kept similar to the way these language models are usually pre-trained, by randomly masking part of sentences.
Let’s now define our Sequential Integrated Gradients (SIG) method. For a word xi and a feature j, SIG is defined as:
SIGij(x) := (xij − xij)×∫ 1
0 ∂F(xi + α× (x − xi)) ∂xij dα
Similar to the original IG (Sundararajan et al., 2017), we compute the gradient of F along a straight line between xi and x for each word xi, the main difference being that the baseline differs for each word. Also similar to the original IG, we approximate in practice the integral with Riemann summation.
Finally, we compute the overall attribution of a word by computing the sum over the feature dimension j, and normalising the result:
SIGi(x) := ∑
j SIGij ||SIG||
Axioms satisfied by SIG The original Integrated Gradients method satisfies a few axioms that are considered desirable for any explanation methods to have. Among these axioms, SIG follows implementation invariance, which states that attributions should be identical if two models are functionally equivalent. Moreover, SIG follows completeness
2Certain language models, such as GPT-2 (Radford et al., 2019), do not have a "mask" token. A "pad" token should be therefore used for such models.
in a specific way: for each word xi, we have the following result:
∑
j
SIGij(x) = F(x)− F(xi)
This means that for each word, the sum of its attribution across all features j is equal to the difference between the output of the model as x and at its corresponding baseline xi. However, it does not entail that ∑ ij SIGij(x) = F(x)− F(x), where x would be an overall baseline filled with <mask>. Moreover, this last axiom entail another one called sensitivity, which here means that if, for a certain word, the input x has the same influence on the output of F as its corresponding baseline xi, then ∑ j SIGij(x) = 0.
Finally, we show in Appendix A that SIG preserves symmetry for each word on the embedding dimension, but that this axiom is not true in general.
Using mask instead of pad as a baseline We propose in this study to replace, as the baseline, the commonly used "pad" token with the "mask" token, on language models having such token. This seems to go against the intuition that the baseline should be uninformative, as "mask" is a trained
token. To support the usage of "mask", we argue that, because <PAD> (denoting the embedding of "pad") is untrained, it could be arbitrarily close to some words, and far from others. Oh the other hand, <MASK> has been trained to replace random words, making it ideally as close to one word as to any other.
Another way to see it is to compare it with images. It is natural for images to choose the baseline as a black image, as this baseline has no information. However, there is no such guarantee in NLP. For instance, the embedding of "pad": <0, 0, 0, . . . , 0> could perfectly be very close to an embedding of a word with a specific meaning, which would harm the explanations. On the other hand, <MASK> has been trained to replace any word, and therefore seems more suited to be the baseline.

heading: 3 Experiments

heading: 3.1 Experiments design
text: We evaluate SIG against various explanation methods by closely following the experimental setup of Sanyal and Ren (2021). As such, we use the following language models: BERT (Devlin et al., 2018), DistilBERT (Sanh et al., 2019) and RoBERTa (Liu
et al., 2019). We also use the following datasets: SST2 (Socher et al., 2013), IMDB (Maas et al., 2011) and Rotten Tomatoes (Pang and Lee, 2005), which classify sentences into positive or negative sentiments or reviews. Moreover, we use the HuggingFace library to recover processed data and pretrained models (Wolf et al., 2019).
Following (Sanyal and Ren, 2021), we use the following evaluation metrics: Log-Odds (Shrikumar et al., 2017), Comprehensiveness (DeYoung et al., 2019) and Sufficiency (DeYoung et al., 2019). These metrics mask the top or bottom 20 % important features, according to an attribution method,
and measure by how much the prediction of the language model changes using this masked data, compared with the original one. For more details on these metrics, please see Sanyal and Ren (2021).
Finally, we use the following feature attribution methods to compare our methods against: Grad*Inp (Shrikumar et al., 2016), Integrated Gradients (Sundararajan et al., 2017), DeepLift (Shrikumar et al., 2017), GradientShap (Lundberg and Lee, 2017) and Discretized IG (DIG) (Sanyal and Ren, 2021) using the GREEDY heuristics. Moreover, as in Sanyal and Ren (2021), we use 50 interpolation steps for all methods expect from DIG, for which we use 30 steps.

heading: 3.2 Results
text: Comparison with other feature attribution methods We present of Tables 1, 2 and 3 a comparison of the performance of SIG with the attribution methods listed in 3.1. We observe that SIG significantly outperforms all other methods across most datasets and language models we used. This tends to confirm that the change of overall meaning of a sentence by combining interpolations simultaneously is an important issue which needs to be tackled.
Comparison between IG and DIG Although results in Sanyal and Ren (2021) show that DIG outperforms other methods, including IG, this is not the case when using "mask" as a token. This result seems to undermine the intuition of Sanyal and Ren (2021) that the discrete nature of the embedding space is an important factor when explaining a language model. We also show in Appendix C that the requirement of having a monotonic path, stressed by Sanyal and Ren (2021), is not necessary.
Choice of the baseline token We also provide in Appendix B results using "pad" as a baseline. Comparison between Tables 1, 2 and 3 on one hand, and Tables 6, 7, 8 on the other hand show that IG greatly improves using the "mask" token as a baseline. This seems to confirm our intuition of using this token instead of "pad". Moreover, SIG performs similarly using either token, which demonstrates the robustness of this method across these two baseline tokens.
Time complexity of SIG One important drawback of SIG is its time complexity, which is dependent on the number of words in the input data. In Table 4, we compare the original IG with SIG, using different numbers of steps. We define t and t′ as the time complexity of computing IG with respectively 50 and 250 steps, and N the number of words in the input data. This table shows that, although reducing the number of steps results in a decrease of performance, SIG with 10 steps still performs better than both IG with 250 steps and IG with 10 × N steps, while having the same time complexity.
Moreover, as noted in Sanyal and Ren (2021), using IG with a large number of steps decreases Delta = ∑ ij IGij(x) − (F(x) − F(x)), while not improving performance. As a result, when computing attributions on long sentences or large texts, we recommend using SIG with a reduced number of steps instead of IG.
Comparison of IG and SIG on several examples We provide on Table 5 several examples of explained sentences, using IG and SIG. Both methods tend to agree on short sentences, while more disagreements appear on larger ones. For each example, we display in underlined bold the most
important token, and in bold the top 20 % most important tokens, according to each method.

heading: 4 Conclusion
text: In this work, we have defined an attribution method specific to text data: Sequential Integrated Gradients (SIG). We have shown that SIG yields significantly better results than the original Integrated Gradients (IG), as well as other methods specific to language models, such as Discretized Integrated Gradients (DIG). This suggests that keeping the meaning of interpolated sentences close to the original one is key to producing good explanations. We have also shown that, although SIG can be computationally intensive, reducing the number of interpolations still yields better results than IG with a greater number of interpolations.
We have also highlighted in this work the benefit of using the token "mask" as a baseline, instead of "pad". Although SIG seems to be robust across both tokens, this is especially important when using IG, as it significantly improves the quality of explanations. Using the trainable token "mask" is indeed closer to the training procedure of language models, and should yield better interpolations as a result. We recommend therefore using this token as a baseline, when possible, when explaining predictions made by a language model.
Moreover, while this study was conducted on bidirectional language models such as BERT, SIG could also be used on auto-regressive models such as GPT-2 (Radford et al., 2019), by iteratively computing the attribution of a token, while keeping previous tokens fixed, and masking future tokens if any has been already computed.
Limitations
We see two main limitations of this work. The first one concerns the diversity of the language models and datasets used. BERT, DistilBERT and RoBERTa have similar architecture, and SST2, IMDB and Rotten Tomatoes are datasets designed to evaluate the sentiment of English text. It would therefore be interesting to validate the robustness of our results on more diverse languages, tasks and language models. In this short paper, we decided for brevity to follow the experiment design of Sanyal and Ren (2021), while being aware of its inherent limitations.
The second limitation of this work concerns the time complexity of SIG. As it needs to compute explanations for each word individually, this method can become very computationally expensive when applied on large text data. To alleviate this issue, we first made it possible to compute gradients in parallel, using an internal batch size similar to how Captum (Kokhlikyan et al., 2020) implemented the Integrated Gradients method. Secondly, as discussed in 3.2, it is possible to reduce the number of interpolated points, which makes the computation faster while retaining better performance than the original IG.
In this work, we ran our experiments on a machine with 16 CPUs, and one Nvidia Tesla T4 GPU. With this setting, computing SIG on SST2 and Rotten Tomatoes takes around one hour for each model. On the larger IMDB, computing SIG, on 2000 randomly sampled inputs, takes around 5 days for BERT and RoBERTa, and 2 days for DistilBERT.
Ethics Statement
The methods presented in this work aim to explain language models, and can as such present ethical issues related to this task. Discriminating biases can indeed be present in text data on which a language model is trained, and such a model can acquire and propagate these biases (Sap et al., 2019). As the presented methods aim to explain a language model without additional knowledge, these methods could also display discriminating biases learnt by a language model.
Moreover, common explanation methods such as Integrated Gradients has proved to be prone to adversarial attacks (Dombrowski et al., 2019), and can be misleading when used on out of sample data (Slack et al., 2021). There is no reason to believe our methods would be more robust compared to
existing methods such as IG. The proposed methods can also be characterised as gradient-based, as they rely on computing gradients on the input data, an uninformative baseline, or on interpolated points between them. As noted by (Mittelstadt et al., 2019), such methods are only local and may not give a clear explanation of the model globally.

heading: Acknowledgement
text: The author would like to thank Vitalii Zhelezniak for his thoughtful comments and suggestions, including using the "mask" token as a baseline. We also thank Anthony Hu for his detailed initial review of this paper.

heading: A On the symmetry-preserving axiom of
text: Sequential Integrated Gradients
This section is divided into two parts. First, we show that SIG preserves symmetry for each word along the embedding dimension. Second, we provide a counterexample to show that symmetry does not hold in general.
Symmetry of SIGi Let us use the same notations as in Section 2. We want to compute the attribution of a word xi on a model F, using the baseline xi. Let’s define the function:
Fi(x) := F(x1, ..., x, ..., xm)
Fi corresponds to F where only the ith word is not fixed. Here, x corresponds to a word, not a sentence.
For such a function, SIG corresponds to the regular IG method: the baseline is < mask > and SIG constructs a straight line between this baseline and xi. As a result, if Fi is symmetric on two embedding features j1 and j2, SIG preserves this symmetry: SIGij1(x) = SIGij2(x).
Non symmetry of SIG The fact that SIG does not preserve symmetry in general is due to the choice of the baseline. As a counterexample, let’s define a language F which takes as an input two words x1 and x2. This language model is moreover symmetric: F(x1, x2) = F(x2, x1).
Here, the original IG method would preserve the symmetry: as the baseline is (<mask>, <mask>), when x1 = x2, we have IG(x)1 = IG(x)2. However, SIG doesn’t preserve the symmetry due to its baseline: we would have: x1 = (<mask>, x2) and x2 = (x1, <mask>). As a result, SIG(x)1 = SIG(x)2 only if x1 = x2 = <mask>.

heading: B Additional results using the "pad" token
text: We present in this section results using the "pad" token instead of the "mask" one. These results for the three datasets: SST2, IMDB and Rotten Tomatoes can be found respectively on Tables 6, 7 and 8.
When using the "pad" token as a baseline, SIG seems to perform similarly compared with using the "mask" one, while other methods perform significantly worse. This demonstrates both the need to use "mask" as a token, and the robustness of the SIG method across different baselines.

heading: C Challenge of the monotonic assumption of the path
text: Sanyal and Ren (2021) stipulate that the path between a baseline and an input needs to be monotonic to allow approximating the integral in IG using Riemann summation. However, while this is true for a Riemann integral, it is also possible to approximate the Riemann–Stieltjes integral, which is a generalisation of Riemann integral, and does not need a monotonic path. We define the Riemann–Stieltjes integral of f : [a,b] → R as:
∫ b
x=a f(x) dg(x)
where g : [0, 1] → [a,b] designates a path. Let us define a partition over [0, 1] as tk such as 0 ≤ t1 ≤ ... ≤ tn ≤ 1. We can then approximate the integral with the sum:
n−1∑
i=0
f(g(ci))× [g(ti+1)− g(ti)]
where ci ∈ [ti, ti+1]. As such, while the partition ti, i ∈ {1, ..., n} needs to be monotonic, the function g does not need to have this constraint. As a result, we could define a path-based IG method as:
IGγ(x)i := ∫
γ ∂F(x) ∂xi dxi
where γ is not necessarily monotonic. (Lundstrom et al., 2022) provide more insights on this topic, and in particular show that the implementation invariance, completeness and sensitivity axioms hold for non-monotonic paths.
For this reason, we decided not to include a combination of DIG and SIG in this study. However, an implementation of this method and the corresponding results can be found in the repository published with this paper.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section at the end of the paper, before references
3 A2. Did you discuss any potential risks of your work? In the introduction and limitation sections
3 A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Left blank.
3 B1. Did you cite the creators of artifacts you used? I created my own and used code from https://github.com/INK-USC/DIG
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Open source
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? In the appendix
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. In the experiment section
C 3 Did you run computational experiments? In the experiment section
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? The models used are pre-trained, standard language models. The computation budget and infrastructure used is discussed in the ethics statement.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Not applicable. The is no training and hyperparameter search
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? It is just a single run - the results are not stochastic
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: , 2019), by iteratively computing the attribution of a token, while keeping previous tokens fixed, and masking future tokens if any has been already computed. Limitations
We see two main limitations of this work. The first one concerns the diversity of the language models and datasets used. BERT, DistilBERT and RoBERTa have similar architecture, and SST2, IMDB and Rotten Tomatoes are datasets designed to evaluate the sentiment of English text. It would therefore be interesting to validate the robustness of our results on more diverse languages, tasks and language models. In this short paper, we decided for brevity to follow the experiment design of Sanyal and Ren (2021), while being aware of its inherent limitations. The second limitation of this work concerns the time complexity of SIG. As it needs to compute explanations for each word individually, this method can become very computationally expensive when applied on large text data. To alleviate this issue, we first made it possible to compute gradients in parallel, using an internal batch size similar to how Captum (Kokhlikyan et al. , 2020) implemented the Integrated Gradients method. Secondly, as discussed in 3. 2, it is possible to reduce the number of interpolated points, which makes the computation faster while retaining better performance than the original IG. In this work, we ran our experiments on a machine with 16 CPUs, and one Nvidia Tesla T4 GPU. With this setting, computing SIG on SST2 and Rotten Tomatoes takes around one hour for each model. On the larger IMDB, computing SIG, on 2000 randomly sampled inputs, takes around 5 days for BERT and RoBERTa, and 2 days for DistilBERT