SYNTAXSHAP: Syntax-aware Explainability Method for Text Generation (2024)

Abstract: To harness the power of large language models in safety-critical domains, we need to ensure the explainability of their predictions. However, despite the significant attention to model interpretability, there remains an unexplored domain in explaining sequence-to-sequence tasks using methods tailored for textual data. This paper introduces SyntaxShap, a local, modelagnostic explainability method for text generation that takes into consideration the syntax in the text data. The presented work extends Shapley values to account for parsing-based syntactic dependencies. Taking a game theoric approach, SyntaxShap only considers coalitions constraint by the dependency tree. We adopt a model-based evaluation to compare SyntaxShap and its weighted form to state-ofthe-art explainability methods adapted to text generation tasks, using diverse metrics including faithfulness, coherency, and semantic alignment of the explanations to the model. We show that our syntax-aware method produces explanations that help build more faithful and coherent explanations for predictions by autoregressive models. Confronted with the misalignment of human and AI model reasoning, this paper also highlights the need for cautious evaluation strategies in explainable AI.1

heading: 1 Introduction
text: Language model (LM) interpretability has become very important with the popularity of generative AI. Despite the great results achieved by the most recent LMs, there is still a large range of tasks where the models fail, e.g., capturing negations (Truong et al., 2023). Therefore, it is crucial to get a better understanding of the LM reasoning and develop faithful explainability methods. As many LMs have little transparency and their use is restricted to API calls, model-agnostic explainability methods have
1Code is publicly available here and on the SyntaxShap website.
become the most practical techniques for gaining better insights into LMs.
The SHapley Additive exPlanations (SHAP) framework is popular for generating local explanations thanks to its solid theoretical background and general applicability (Shapley et al., 1953). However, regarding sequence-to-sequence tasks such as next token generation, the usage of SHAP-based methods has not been explored in depth (Mosca et al., 2022). We address this gap and develop a coalition-based explainability method inspired by Shapley values for text generation explanation.
Our explainability method (in Figure 1) considers syntactic word dependencies (de Marneffe et al.). The syntax is important as next-word predictions in autoregressive (AR) LMs underlie implicit incremental syntactic inferences, i.e., LMs implicitly capture dependencies in text data (Eisape et al., 2022). In this paper, we investigate if dependency parsing trees can be used in the explainability process as syntactic relational graphs and help shed light on the influence of words on the model’s prediction given their syntactic role in the sentence.
We evaluate the explanations on diverse metrics. First, we adapt fidelity, one of the most popular model-based evaluation metrics in xAI (eXplainable AI), to the text generation task and introduce two new metrics to test whether the generated explanations are faithful to the underlying model. Second, we introduce two qualitative evaluation metrics that capture the explanation quality with regard to human expectations, i.e., the coherency of explanations and their semantic alignment. Our evaluation procedure compares our method SyntaxShap to state-of-the-art explainability methods. Explanations produced by our method of the next token generation by two popular AR models respect the sentence syntactic structure while being faithful and more coherent compared to state-ofthe-art SHAP-based methods that do not explicitly consider the word dependency for text generation
4551
tasks. Our semantic alignment analysis also proves that faithful explanations do not necessarily meet human expectations in terms of token importance, highlighting a clear disagreement between our human mental model and AI models.
To summarize, our contributions are (1) SyntaxShap, a new SHAP-based explainability method that incorporates dependency tree information, (2) quantitative metrics that address LM’s stochasticity and qualitative metrics to account for human semantic expectations, and (3) an evaluation of the explanation quality on two AR LMs. Our work opens multiple research directions for future work.

heading: 2 Related Work
text: Explainability in Linguistics Syntax and semantics play an important role in explaining LM outcomes from a linguist perspective. Multiple attempts were made to explore the role of syntactic and semantic representations to enhance LM predictions. Ek et al. (2019) look at the role of syntactic and semantic tags for the specific task of human sentence acceptability judgment. They show that syntactic tags significantly influence the predictions of the LM. In recent years, there has been an increasing interest in methods that incorporate syntactic knowledge into Machine Translation (Ambati, 2008). In addition, Eisape et al. (2022) has shown that next-word predictions from AR neural LMs show remarkable sensitivity to syntax. However, there has been no attempt to account for the syntax in explanations of those LMs for text generation tasks (Mosca et al., 2022). For this reason, we propose to incorporate syntax-based rules
to explain AR LM text generation.
SHAP-based explainability in NLP One way to categorize model-agnostic post-hoc explainability methods is to separate them into perturbation-based and surrogate methods (Zhao et al., 2023). Among the most popular surrogate models are LIME and SHAP. The Shapley-value approach (Shapley et al., 1953) provides local explanations by attributing changes in predictions for individual data inputs to the model’s features. Those changes can be combined to obtain a better global understanding of the model structure. For text data, available approaches seem mostly tailored to classification settings (Kokalj et al., 2021; Chen et al., 2020).
Shapley values and complex dependencies One underlying assumption of SHAP is feature independence. Confronted with more diverse types of data inputs, newer methods offer the possibility to account for more complex dependencies between features. Frye et al. (2020) propose Asymmetric Shapley values, which drop the symmetry assumption and enable the generation of model-agnostic explanations incorporating any causal dependency known to be present in the data. Following this work, Heskes et al. (2020) propose Causal Shapley values to account more specifically for causal structures behind feature interactions. Chen et al. (2019) construct coalitions based on a graph structure, grouping features with their neighbors or connected nodes. When it comes to text data, words present strong interactions, and their contribution heavily rely on the context. Therefore, feature attributions for textual data should be specifically tai-
lored to account for those complex dependencies. HEDGE is one example of a SHAP-based method addressing the context dependencies specific to text data (Chen et al., 2020). It hierarchically builds clusters of words based on their interactions measured by the cohesion score. While their objective is to cluster words to minimize the loss of faithfulness, i.e., prediction change, we propose a new strategy to create coalitions of words that respect the syntactic relationships dictated by the dependency tree. This way, we consider the syntactic dependencies that are the basis of linguistics and which were proven essential for next-word predictions from AR LMs (Eisape et al., 2022).

heading: 3 SyntaxShap Methodology

heading: 3.1 Objective
text: Given a sentence of n words x = (x1, ..., xn) and ŷ = (ŷ1, ..., ŷm) the m generated words by an AR LM f , the objective is to evaluate the importance of each input token for the prediction ŷ. We focus on explaining the next token, i.e., m = 1. Let fy(x) be the model’s predicted probability that the input data x has the next token y. Our method produces local explanations.

heading: 3.2 Shapley values approach
text: We adopt a game theory approach to measure the importance of each word xi to the prediction. The Shapley value approach was first introduced in cooperative game theory (Shapley et al., 1953) and computes feature importance by evaluating how each feature i interacts with the other features in a coalition S. For each coalition of features, it computes the marginal contribution of feature i, i.e., the difference between the importance of all features in S, with and without i. It aggregates these marginal contributions over all subsets of features to get the final importance of feature i.

heading: 3.3 Syntax-aware coalition game
text: Our work focuses on incorporating syntax knowledge into model-agnostic explainability. We adopt a coalition game approach that accounts for these syntactic rules. As illustrated in Figure 1, SyntaxShap computes the contribution of words only considering allowed coalitions S constraint on the dependency tree structure. We define a coalition S as a set of words or features {xi, i ∈ [1, n]} from the input sentence x. Given a dependency tree with L levels, li ∈ [1, L] corresponds to the level of
word xi in the tree and nl > 0 the number of words at level l in the tree. To compute the contribution of the words in the tree, SyntaxShap only considers the allowed coalitions S = ⋃L l=0Sl, where Sl is the set of allowed coalitions at level l. We pose the default S0 = {S0} and S0 = {} is the null coalition. Notations Let Xl be the set that contains all the words at level l, X<l the one that contains all the words before level l in the tree, and P(Xl) the powerset, i.e. the set of all subsets of Xl. Definition (Set of coalitions at level l) The set of coalitions Sl at level l is defined as:
Sl = ⋃
σ∈P(Xl) X<l ∪ σ
Property At each level of the tree, each coalition S ∈ Sl respects two properties:
∀i ∈ [1, n] s.t. li > l, xi /∈ S. (1) ∀i ∈ [1, n] s.t. li < l, xi ∈ S. (2)
Given the tree-based coalitions, we can compute the contribution of each token in the input sentence to the model’s prediction. The contribution of feature xi at level li on the dependency tree to the model output ŷ is defined as:
ϕi = 1
Ni
∑
S∈ (
li−1⋃ p=0 Sp
) ⋃
S \i li
[fŷ(S ∪ {xi})− fŷ(S)] (3)
where Ni corresponds to the number of allowed coalitions at level li that do not contain feature xi, and S\il corresponds to the set of coalitions at level l that exclude word xi, i.e.,
S \i l =
⋃
σ∈P(Xl) X<l ∪ (σ\{xi}).
Property Given the number nl of words (or nodes) at level l of the tree, each word at the same level shares the same number of updates, i.e., allowed coalitions, i.e., ∀xi s.t. li = l, Ni = N l and Nl can be expressed as:
Nl = l−1∑
p=0
2np + 2nl−1 − l (4)
Proof To compute Nl in equation Equation 4, we proceed recursively starting from the root nodes. The dependency has L levels starting from level
l = 1. We postulate a hypothetical level 0 where the null coalition S0 = {} can be formed. At level 1, there is the root node of the tree, i.e. n1 = 1. The number of coalitions is |S1 = {{xroot}}| = 1. Let nl be the number of nodes at level l. The number of combinations of nl features is 2nl . Since we already counted the null coalitions at the hypothetical level 0, we don’t count it in the allowed coalitions Sl at level l. We arrive at the final number of coalitions |Sl| = 2nl−1. Now, let’s say we have a word x at level l. This word can join all allowed coalitions at level < l — there are 1+ ∑l−1 p=1(2
np−1) — and all the coalitions of the words at level l where x does not appear — there are 2nl−1−1. In conclusion, we find that the number of allowed coalitions for word x at level l is:
Nl = 1 +
l−1∑
p=1
(2np − 1) + 2nl−1 − 1
= l−1∑
p=0
2np + 2nl−1 − l
We pose n0 = 0, the number of nodes on the hypothetical level 0, to start the sum at p = 0 for simplification.
Our strategy of building tree-based coalitions drops the efficiency assumption of Shapley values but preserves the symmetry axioms for the words at the same level of the dependency tree, as well as the nullity and additivity axioms. Appendix subsection B.1 details the four shapley axioms and discusses which ones SyntaxShap respects or violates. Note that this does not undermine the quality of the explanations since the axioms were shown to work against the goals of feature selection in some cases (Fryer et al., 2021).

heading: 3.4 Weighted SyntaxShap
text: In the context of text data and syntactic dependencies, we assume that words at the top of the tree should be given more importance since they are the syntactic foundations of the sentence and usually correspond to the verb, subject, and verb qualifiers. Therefore, we propose SyntaxShap-W, a variant of our method that weighs words according to their position in the tree. The weights are treelevel-dependent and correspond to the inverse of the word level for which contribution is computed, i.e., wl = 1/l. The contribution of a word xi at
level li can be expressed as:
ϕi = wli Ni
∑
S∈ (
li−1⋃ p=0 Sp
) ⋃
S \i li
[fŷ(S ∪ {xi})− fŷ(S)] (5)

heading: 4 Evaluation
text: This section describes our model-based evaluation procedure that encompasses both quantitative and qualitative analysis of the explanations. While previous works only focus on the faithfulness of explanations to assess their quality, we also propose to consider human qualitative expectations.

heading: 4.1 Quantitative evaluation
text: To analyze if the explanations are faithful to the model, we adopt fidelity, the most common modelbased metric in xAI (Carvalho et al., 2019), which looks at the top-1 prediction and propose two new variants that balance the LM’s probabilistic nature by considering the top-K predictions. By considering the first top K predictions, we balance the explainer’s deterministic nature and the inherent stochasticity of language models.
Fidelity Fidelity measures how much the explanation is faithful to the model’s initial prediction for the next token. By keeping the top t% words in the input sentence, fidelity calculates the average change in the prediction probability on the predicted word over all test data as follows,
Fid(t) = 1
N
N∑
i=1
(fŷ(xi)− fŷ(x̃(t)i )) (6)
where x̃(t)i is the masked input sentence constructed by keeping the t% top-scored words of xi, ŷ is the predicted token given input xi, i.e. ŷ = argmaxy′ fy′(xi), and N is the number of examples. Usually, the missing words are replaced by the null token, but we also propose an alternative fidelity Fidrand by replacing the missing words with random words from the tokenizer vocabulary.
Probability divergence@K The probability divergence at K corresponds to the average difference in the top K prediction probabilities on the predicted class over all test data. It can be expressed as follows,
div@K = 1
N
N∑
i=1
K∑
k=0
(fŷk(xi)− fŷk(x̃i(t))) (7)
where ŷk is the top kth prediction given input xi. We choose K = 10 because most of the sentences can be completed with multiple possible words that are synonyms or semantically consistent with the input sentence.
Accuracy@K The accuracy at K corresponds to the average ratio of common top K predictions between the full and masked sentences:
acc@K = 1
N
N∑
i=1
∣∣∣{ŷk, k≤K} ∩ {ỹ(t)k , k≤K} ∣∣∣
K
(8)
where ỹ(t)k is the top k th prediction given input x̃(t)i .

heading: 4.2 Qualitative evaluation
text: Coherency Coherency describes how similar the explanation is w.r.t. similar next generated token. In other words, given a pair of input sentences with a slight variation in the syntax but a strong change in semantics (e.g., differing only by a negation), we expect similar explanations for similar model’s predictions and dissimilar ones when the model is sensitive to the perturbation.
Semantic alignment An important criterion to evaluate a textual explanation is whether it is aligned with human expectations. As humans, we intuitively expect the language model to draw little attention to tokens in the input sentence, which semantic substance is not reflected in the prediction. This semantic alignment can be measured for some semantically rich tokens that are decisive for text generation, e.g., the negation. Given a decisive token in input sentences and a model’s prediction that does not semantically account for it, we compare methods on the importance rank attributed to this token. An explainability method is semantically aligned if this rank is high, i.e., the decisive token is not important for the model’s prediction.

heading: 5 Experiments
text: We evaluate SyntaxShap and SyntaxShap-W on various criteria such as faithfulness in section subsection 5.2, the coherency in section subsection 5.3, and the semantic alignment of their explanations in section subsection 5.4.

heading: 5.1 Experimental setting
text: For the evaluation, we use three datasets, i.e., the Generics KB2 (Generics) (hug, 2020), ROCStories
2published at the ACL Anthology, CC BY 4.0 License
Winter20173 (ROCStories) (Mostafazadeh et al., 2017), and Inconsistent Dataset Negation2 (Negation) (Kalouli et al., 2022). They have the following characteristics: (1) The Generics dataset contains high-quality, semantically complete statements; (2) The ROCStories dataset contains a collection of five-sentence everyday life stories; (3) The Negation dataset contains disjoint sentence pairs, i.e., a sentence and its negated version. For evaluation purposes, we first separate the stories of the ROCStories dataset into single sentences and remove the last token from sentences in the three datasets. We use the TextDescriptives component in spaCy to measure the dependency distance of the analyzed sentences following the universal dependency relations established by de Marneffe et al. and compute the average number of tokens per sentence as well as the number of unique tokens in the three datasets. As shown in Table 2, sentences in the Generics and ROCStories datasets have more complex syntactic structures, and the sentences are longer than in the Negation dataset. However, the Negation dataset includes sentences with minimal syntactic variation but great semantic differences, enabling fine-grained qualitative analysis. This makes it the most suitable dataset for comparing xAI methods on coherency and semantic alignment.
To assess the performance of our method, we use two AR LMs: GPT-2 model (Radford et al., 2019) consisting of 154M parameters 4 and Mistral 7B (Jiang et al., 2023) with 7B parameters 5. This significant increase in parameters allows Mistral 7B to generate next tokens that are more contextualized and semantically rich. Mistral 7B was also shown superior to other 7B LMs and therefore much more advanced than GPT-2 (Jiang et al., 2023). For this reason, we favor Mistral 7B for the qualitative analysis, as the predictions are more specific and better aligned with the context of the input sentences. We reproduce our experiments on four different seeds and convey the mean and variance of our results. Our methods SyntaxShap and SyntaxShap-W are compared against the Random baseline, i.e. a normally distributed token importance attribution, and two other explainability baselines LIME (Ribeiro et al., 2016) and FeatureAblation, a perturbationbased method that replaces each input feature with
3publicly available, no license 4GPT-2 model was taken from HuggingFace https:// huggingface.co/openai-community/gpt2 5Mistral 7B model was taken from HuggingFace https: //huggingface.co/mistralai/Mistral-7B-v0.1
experiments on four different random seeds. The methods introduced in this paper are shaded.
a given baseline and computes the difference in output. FeatureAblation’s individual token substitutions may not fully reflect feature interactions. We also compare to SampleShapley, an approximation of SHAP that computes the contribution of each input token considering random permutations of the input features. All baselines have been adapted for text data and LLMs by defining interpretable text features and handling sequential predictions. We also compare them against Partition, a faster version of KernelSHAP that hierarchically clusters features. Its implementation is based on HEDGE (Chen et al., 2020), a SHAP-based method that builds hierarchical explanations via divisive generation, respecting some pre-computed word clustering, and is particularly suited for text data6.
To derive textual explanations from the explanatory masks, we opted for consistency with the SHAP Python library by adopting attention mask modification. This method involves setting the attention weight of a token to zero to simulate its removal from the input data. In our evaluation with faithfulness metrics, we experimented with various strategies, including modifying attention weights and replacing tokens with random selections from the tokenizer vocabulary. However, we found minimal differences in the results across
6The code for FeatureAblation and SampleShapley was taken from the Captum Python library, and for Partition from the SHAP Python library. LIME was adapted for text data from its initial implementation.
these approaches. For a detailed comparison of these masking strategies, we direct readers to the Appendix subsection D.1.

heading: 5.2 Faithfulness
text: In this section, we evaluate the faithfulness of our explanations to Random, LIME, FeatureAblation, SampleShapley, Partition, and our methods SyntaxShap and SyntaxShap-W on the full datasets with sentence lengths between 5 and 15 tokens.
For both models, Mistral 7B in Figure 2 and GPT-2 in Figure 3, our methods SyntaxShap and SyntaxShap-W produce more faithful explanations than the trivial random algorithm, the LIME method adapted to NLP tasks, and Partition, the state-of-the-art shapley-based local explainability method for text data. Therefore, building coalitions based on syntactic rules gives more faithful explanations than when minimizing a cohesion score, preserving the strongest word interactions (Chen et al., 2020). For the Mistral 7B model, they also both outperform the leave-one-out method, FeatureAblation, and the approximation-based shapley values, SampleShapley, which produce poorly faithful explanations of Mistral 7B predictions. For GPT-2 predictions on the Generics and ROCStories datasets in Figure 9a and Figure 9b, SyntaxShap(W) still lags slightly behind FeatureAblation and SampleShapley. However, while generating more faithful explanations for GPT-2 model, FeatureAblation and SampleShapley’s ablate tokens independently, overlooking highly correlated tokens within a sentence (Miglani et al., 2023).
SyntaxShap(-W) generates more faithful explanations than the random baseline, LIME, and Partition. Although it does not beat the baselines FeatureAblation and SampleShapley for every

heading: 5.3 Coherency
text: In this section, we explore whether SyntaxShap produces coherent explanations with the model understanding. For this evaluation, we use Mistral 7B and run a perturbation analysis using sentence pairs from the Negation dataset. We use a sample of 267 sentence pairs (with and without the negation not and with varying usage of with and without) whereby for 90 pairs, the model predicts the same next token. An example of two sentence pairs is shown in Figure 4. For pairs with equal predictions (e.g., A mom is not a and A mom is a with an equal next token prediction super), we expect more similar attribution ranks than for pairs with different predictions (e.g., A person has no right and A person has died). To evaluate the coherency,
we first represent the attribution scores as rank vectors. We then separate pairs with equal predictions and different predictions into two distinct groups and measure the cosine similarity between rank vectors of each pair within each group, whereby negation words are excluded to get equal-length vectors. The average difference in cosine similarity between the two groups for each explainability method is displayed in Figure 5. It shows that SyntaxShap produces more similar attributions for
sentence pairs that predict the same next token and more diverse attributions for sentence pairs with different next token predictions.
Given a pair of sentences with and without a negation, which theoretically have two disjoint semantic meanings, the similarity of SyntaxShap’s token attribution values for each sentence better reflect the degree of similarity of the next token predictions than LIME, FeatureAblation, SampleShapley and Partition.

heading: 5.4 Semantic alignment
text: Here, we explore whether the generated explanations are aligned with human semantic expectations. We analyze cases where there is a negation in a sentence, but the model’s prediction does not reflect it, e.g., A father is not a father. We extract negative instances, i.e., that contain the token not, no, or without, from the Negation dataset. We label those where the language model predicts wrong next tokens, i.e., semantically misaligned with the negation. The average importance scores of the negation tokens in each of the 22 labeled instances for Mistral 7B are reported in Figure 6. Following our human mental model, we expect a low importance score assigned to the negation token if the model produces predictions that do not account for it. However, Figure 6 shows that SyntaxShap(W) identifies negation as the most important token in more than 80% of the cases. This indicates a misalignment between Mistral 7B’s reasoning and our mental model. Furthermore, SyntaxShap(W) produces the most faithful explanations for the Negation dataset with Mistral 7B (see Figure 2). This suggests that faithful explanations do not match human expectations. The misalignment between human and AI reasoning supports the belief that model-focused explanations, which are faithful to the model, are not necessarily intended for human interpretation. Therefore, the evaluation of explanations depends on the focus: human-focused explanations must meet human expectations, while model-focused explanations are intended to be faithful to the language model.
Explainability methods, although faithful, can assign low-importance scores to tokens that humans consider decisive for predictions, revealing a fundamental misalignment between human and AI model reasoning.

heading: 6 Discussion
text: Addressing stochasticity The traditional faithfulness metrics like fidelity, AOPC (Samek et al., 2016; Nguyen, 2018) or log-odds (Shrikumar et al., 2017; Chen et al., 2019) scores take a deterministic approach to evaluate explanations computed on stochastic predictions. This paper evaluated AR LMs that adopt top-k sampling to randomly select a token among the k tokens with the highest probability. To account for this stochasticity, we proposed additional evaluation metrics, div@K and acc@K, that consider not only the final prediction but the top-K predictions, balancing the model’s probabilistic nature. Nevertheless, further methods that address the stochastic nature of the models should be designed in future research.
Integrating linguistic knowledge To ensure that the explainability methods produce meaningful explanations that mimic AR LM behavior, we need to go beyond the faithfulness type of evaluation and consider further explainability aspects. In this paper, we study explanations on other dimensions related to semantic interpretation and coherency of explanations. There is potential for more linguistically tailored evaluation methods in the future. The motivation is as follows. The next token prediction task can be seen as a multi-class classification with a large number of classes. The classes have diverse linguistic properties, i.e., tokens have different roles in the sentence, some being more contentand others function-related. We might want to consider these different roles when evaluating the quality of explanations. On the one hand, with controlled perturbations on the input sentences, we can evaluate the role of semantics and syntax on the next token prediction task. On the other hand, when computing the explanation fidelity, we might consider prediction changes from one category of to-
kens (e.g., function words) to another (e.g., content words), giving a more linguistic-aware explanation quality assessment.
Considering humans When designing evaluation methods, we need to consider humans since, ideally, they should understand model behavior from the produced explanations. There is one main concern, though. As prior work has shown (Sevastjanova and El-Assady, 2022), LM explainability can suffer from human false rationalization of model behaviors. We typically expect the explanations to align with our mental models of language. However, LMs learn language differently from humans; thus, explanations can theoretically differ from our expectations. Thus, future work should design evaluation methods that clearly show the importance of the words for the model and the reasons why this importance (potentially) does not align with human expectations.

heading: 7 Conclusion
text: We proposed SyntaxShap - a local, model-agnostic syntax-aware explanability method. Our method is specifically tailored for text data and meant to explain text generation tasks by AR LMs, whose interpretability in that context has not yet been addressed. SyntaxShap is the first SHAP-based method to incorporate the syntax of input sentences by constraining the construction of word coalitions on the dependency trees. Our experimental results demonstrate that SyntaxShap and its weighted variant can improve explanations in many aspects: they generate more faithful and coherent explanations than the standard model-agnostic explainability methods in NLP for advanced AR LMs like Mistral 7B. The semantic alignment analysis also shows that faithfulness and human intelligibility are distinct evaluation criteria that do not necessarily align. This study addresses a pressing and significant issue regarding the explainability of AR models, contributing to an ongoing dialogue in the research community.
Limitations
Multiple sentences In this paper, our analysis is limited to one input sentence because we work on one dependency tree at a time. However, our method can be scaled to text with multiple sentences or a paragraph by breaking it down into multiple dependency trees and running SyntaxShap in parallel. However, by doing this, we might lose sentence correlations.
Incorrect dependency tree Our method heavily relies on the dependency tree, assuming it correctly captures the syntactic relationships between the words. However, the Python module spaCy sometimes generates arguable dependencies from the perspective of linguists, and its accuracy drops when implemented for languages other than English. Therefore, SyntaxShap is, for now, only meant to be used for English grammatically nonconvoluted sentences to limit the uncertainty coming from the construction of the dependency tree.
Ethics Statement
The data and resources utilized in this study are openly available and widely used by numerous existing works. The datasets employed consist of factual statements devoid of subjective judgments or opinions. It is acknowledged that pre-trained LMs, such as GPT-2 and Mistral 7B, may inherently inherit biases as highlighted in previous research (Radford et al., 2019), potentially influencing the generated next token. For example, certain tokens like beautiful may tend to appear more frequently in contexts associated with female characteristics. While the primary objective of this study is to produce explanations that faithfully represent the model’s predictions, it is recognized that these explanations may also carry inherent biases. It is imperative to acknowledge that the results generated by our approach may not always align with human mental models and could potentially be used in applications that have the potential to cause harm.

heading: A Textual data
text: A.1 Text generation Text generation tasks involve predicting the next word in a sequence, like in language modeling, which can be considered a simpler form of text generation. Other tasks may involve generating entire paragraphs or documents. Text generation can also be framed as a sequence-to-sequence (seq2seq) task that aims to take an input sequence and generate an output sequence for machine translation and question answering. AR models like GPT (Generative Pre-trained Transformer) generate text one word at a time in an AR manner, conditioning each word on the previously generated words. In this paper, we focused on the next token generation task given one single sentence as input. We work with factual sentences from Generics and ROCStories datasets, which often expect a semantically rich final token to complete the clause. Multiple predictions are possible, but only a few are correct. Here is an example of a sentence in the Generics dataset: Studio executive is an employee of a film. The GPT-2 model predicts studio as the next token with the random seed 0. We can expect other predictions like company, firm, or corporation. But the number of possibilities is still very limited.
A.2 Dependency parsing Dependency parsing is a natural language processing technique that involves analyzing the grammatical structure of a sentence to identify the relationships between words (de Marneffe et al.). It involves constructing a tree-like structure of dependencies, where each word is represented as a node, and the relationships between words are represented as edges. Each relationship has one head and a dependent that modifies the head, and it is labeled according to the nature of the dependency between the head and the dependent. These labels can be found at Universal Dependency Relations (de Marneffe et al.). Dependency Parsing is a powerful technique for understanding the meaning and structure of language and is used in various applications, including text classification, sentiment analysis, and machine translation. We use the Python module spaCy (version 3.7.2) (Honnibal and Montani, 2017) to generate dependency trees on the input sentences. The number of tokens varies from 5 to 15 tokens for the Generics and ROCStories datasets, producing very diverse and complex
parsing trees. This diversity enriches our analysis and strengthens our results. The code source for dependency parsing was extracted from https: //stackoverflow.com/questions/7443330/ how-do-i-do-dependency-parsing-in-nltk.

heading: B SyntaxShap: characteristics and proofs
text: B.1 SyntaxShap and the Shapley axioms
The four axioms satisfied by Shapley values, i.e., efficiency, additivity, nullity, and symmetry, do not generally provide any guarantee that the computed contribution value is suited to feature selection, and may, in some cases, imply the opposite (Fryer et al., 2021). We define here new axioms for SyntaxShap values since two of the four Shapley axioms cannot be satisfied by tree-constraint values.
Efficiency The evaluation function v(S) in SyntaxShap is the output probability for the predicted next token given the full input sentence, i.e., v(S) = fŷ(S) where ŷ =argmax(f(x)). Because of the non-linearity of LMs, SyntaxShap evaluation function is non-monotonic. It does not necessarily increase if you add more features. For this reason, SyntaxShap does not satisfy the efficiency axiom. This implies that the SyntaxShap values of each word do not sum up to the SyntaxShap value of the whole sentence.
Symmetry SyntaxShap satisfies the axiom of symmetry at each level of the dependency tree. Any two features xi, xj that are at the same level of the dependency tree, i.e., li = lj , play equal roles and therefore have equal SyntaxShap values:
∀i, j s.t. li = lj [∀(S \ {xi, xj})v(S ∪ xi) = v(S ∪ xi)]
=⇒ ϕi = ϕj (9)
Nullity If feature xi contributes nothing to each submodel it enters, then its SyntaxShap value is zero.
[(∀S)v(S ∪ {xi}) = v(S)] =⇒ ϕi = 0 (10)
Additivity Given two models f and g, the SyntaxShap value of those models is a linear combination of the individual models’ SyntaxShap values:
ϕi(f + g) = ϕi(f) + ϕi(g) (11)
B.2 Computational complexity
One advantage of the SyntaxShap algorithm is its faster computation time compared to the naive Shapley values computations. We estimate each algorithm’s complexity by approximating the total number of computation steps, i.e., formed coalitions and updated values, for the original Shapley values computation and our method.
Shapley values computation The Shapley value of feature x requires the 2n−1 coalitions of all features excluding x. As we need to update n features, the total number of updates is n · 2n−1. The computation complexity is, therefore, in O(n2n).
SyntaxShap The SyntaxShap value of feature x at level l requires Nl updates. Considering all the features in the input, the total number of computations is L∑ l=1
nl ·Nl. To approximate this number, we assume the dependency tree is balanced and poses nl = n/L. In this case, Nl can be re-written as:
Nl = l−1∑
p=0
2n/L + 2n/L−1 − l
= l(2n/L − 1) + 2n/L−1
The total number of computations can now be approximated to:
n
L
L∑
l=1
Nl = n
L
L∑
l=1
( l(2n/L − 1) + 2n/L−1 )
= n
L
( L(L+ 1)
2 (2n/L − 1) + L2n/L−1
)
= n(L+ 1) 2 (2n/L − 1) + n2n/L−1
The approximation of the computation complexity in the case of a balanced tree is O(nL2n/L).

heading: C Data preprocessing
text: SyntaxShap relies on the construction of dependency trees that capture the syntactic dependencies in the sentences. Entities in dependency trees are words as defined in the English dictionary. However, language tokenizers sometimes split words into multiple tokens. SyntaxShap supports subtokenization, which involves splitting a word into multiple tokens. When this occurs, SyntaxShap duplicates the word-node in the dependency tree, ensuring that each token is viewed as a distinct node with an identical role. Subtokens receive equal treatment in the analysis process. Table 3 displays the statistics for the three datasets Negation, Generics, and ROCStories, with the initial number of sentences and the explained sentences after filtering. Our filtering strategy removes sentences with more than 15 tokens because of the computational cost, sentences parsed into multiple spans, and sentences containing punctuations !"#$%&'()*+, -./:;<=>?@[\]^_`{}~ given by the Python module string.
Figure 7 displays the length distribution of sentences in each dataset after filtering. Negation dataset contains short sentences with less than 8 tokens. It is also used in our study for experiments on coherency and semantic alignment of explanations. Generics and ROCStories are more complex and realistic. They also have a greater diversity of words and syntactic complexity as identified by the dependency distance and token diversity in Table 2.
1.0
Generics dataset
Rocstories dataset

heading: D Additional results
text: D.1 Masking strategies
This section analyses the impact of selecting different masking strategies when removing unimportant tokens on the faithfulness of explanations. We want to investigate if there is an optimal masking strategy for explainability with text data. In our evaluation with faithfulness metrics, we experimented with two masking strategies, including (i) modifying attention weights and (ii) replacing tokens with random selections from the tokenizer vocabulary. The left figures in Figure 8 and Figure 9 show the faithfulness scores of explanations when unimportant are masked with null attention, while the right figures display the scores when they are replaced with random tokens. For Mistral 7B model, we observe little difference between the two masking strategies. For GPT-2 model, replacing unimportant tokens with random tokens produces slightly less faithful explanations than removing attention from those tokens. This aligns with our understanding of LLM predictions, as their predictions rely on contextualized embeddings. Random to-
kens can alter the sentence context, impacting the meaning of the important tokens in the explanation for the model. However, both masking strategies ultimately yield the same conclusions, as the relative performance of explainability methods remains consistent in both cases.
D.2 Number of tokens and faithfulness
This section analyzes the relationship between the number of tokens in the input sentences and the performance of the explainability algorithms. We vary the number of tokens from 5 to 15 tokens to have at least 50 sentences of the same length for both Generics and ROCStories and have a decent number of inputs to average upon (see the number of tokens distribution in Figure 7). Figure 10 shows that the performance of all methods is robust to the increase in the number of tokens. SyntaxShap can be applied to a diverse range of sentence lengths.
D.3 Dependency distance and faithfulness
This section explores how the faithfulness of explanations varies with respect to the dependency distance of the input sentence. The average de-
pendency distance (ADD) of a sentence is a good indicator of its syntactic complexity. ADD is calculated by summing the distances of all dependencies in the sentence and then dividing that sum by the number of dependencies in the sentence (Oya, 2011). The distance of a single dependency corresponds to the distance between the two words involved in that dependency within the sentence. We use the TextDescriptives component in spaCy to measure the dependency distance of the analyzed sentences following the universal dependency relations established by de Marneffe et al.. We aim to investigate whether higher syntactic complexity correlates with less faithful explanations, meaning that it may be more challenging for explainability methods to generate faithful explanations. Figure 11 shows that the faithfulness of explanations remains constant across all ranges of dependency distance for the Generics dataset. For ROCStories, we observe slightly less faithful explanations when the input sentences exhibit more complex syntactic structures. However, the number of instances with a low or high dependency distance, as reported in Table 4, is too low to conclude, with 3 input sentences with a dependency distance d < 0.5 and 7 with d > 3. Figure 12 also presents the distribution of sentences grouped by dependency distance range
in the ROCStories and Generics datasets. For ROCStories, most input sentences have a dependency distance between 1 and 2. Therefore, observations made for extreme dependency distance ranges in Figure 11 should be interpreted with caution. We do not have enough instances to conclude whether higher dependency distances imply less faithful explanations.

heading: Future Work
text: Nevertheless, further methods that address the stochastic nature of the models should be designed in future research. Integrating linguistic knowledge To ensure that the explainability methods produce meaningful explanations that mimic AR LM behavior, we need to go beyond the faithfulness type of evaluation and consider further explainability aspects. In this paper, we study explanations on other dimensions related to semantic interpretation and coherency of explanations. There is potential for more linguistically tailored evaluation methods in the future. The motivation is as follows. The next token prediction task can be seen as a multi-class classification with a large number of classes. The classes have diverse linguistic properties, i. e. , tokens have different roles in the sentence, some being more contentand others function-related. We might want to consider these different roles when evaluating the quality of explanations. On the one hand, with controlled perturbations on the input sentences, we can evaluate the role of semantics and syntax on the next token prediction task. On the other hand, when computing the explanation fidelity, we might consider prediction changes from one category of to-
kens (e. g. , function words) to another (e. g. , content words), giving a more linguistic-aware explanation quality assessment. Considering humans When designing evaluation methods, we need to consider humans since, ideally, they should understand model behavior from the produced explanations. There is one main concern, though. As prior work has shown (Sevastjanova and El-Assady, 2022), LM explainability can suffer from human false rationalization of model behaviors. We typically expect the explanations to align with our mental models of language. However, LMs learn language differently from humans; thus, explanations can theoretically differ from our expectations. Thus, future work should design evaluation methods that clearly show the importance of the words for the model and the reasons why this importance (potentially) does not align with human expectations