Attention Mechanisms Don’t Learn Additive Models: Rethinking Feature Importance for Transformers
Authors: Tobias Leemann
Alina Fastowski
Felix Pfeiffer
Gjergji Kasneci
Venue: Accepted by TMLR ()

Abstract: We address the critical challenge of applying feature attribution methods to the transformer architecture, which dominates current applications in natural language processing and beyond. Traditional attribution methods to explainable AI (XAI) explicitly or implicitly rely on linear or additive surrogate models to quantify the impact of input features on a model's output. In this work, we formally prove an alarming incompatibility: transformers are structurally incapable of representing linear or additive surrogate models used for feature attribution, undermining the grounding of these conventional explanation methodologies. To address this discrepancy, we introduce the Softmax-Linked Additive Log Odds Model (SLALOM), a novel surrogate model specifically designed to align with the transformer framework. SLALOM demonstrates the capacity to deliver a range of insightful explanations with both synthetic and real-world datasets. We highlight SLALOM's unique efficiency-quality curve by showing that SLALOM can produce explanations with substantially higher fidelity than competing surrogate models or provide explanations of comparable quality at a fraction of their computational costs. We release code for SLALOM as an open-source project online at https://github.com/tleemann/slalom_explanations.

Published in Transactions on Machine Learning Research (12/2024)
Attention Mechanisms Don't Learn Additive Models: Re- thinking Feature Importance for Transformers
Tobias Leemann University of Tübingen, Technical University of Munich
hobres.leeaammeuuuntuuennngn... de
Alina Fastowski Technical University of Munich
alina.fatooskii@uue de
Felix Pfciffer University of Tübingen
feliz. coleeepfeiffeep/rrtoommnnnt com
Gjergji Kasneci Technicol University of Munich
gjergi.knsmeeci@tam.dd
Reviewed on OpenReview:
net/ ptr/oomm? id- yawWa4gWke?
Abstract
We address the critical challenge of applying feature altribution methods to the transformer architecture, which dominates current applications in natural language processing and beyond. Traditional attribution methods to explainable Al (XAI) explicitly or implicitly rely on linear or additive surrogate models to quantify the impact input features on a model's output. In this work, we formally prove an alarming incompatibility: transformers are structurally incapable of representing linear or additive surrogate models used for feature attribution, undermining the grounding of these conventional explanation methodologies. To address this discrepancy, we introduce the Softmax- Linked Additive Log Odds Model (SLALOM), novel surrogate model specifically designed to align with the transformer framework. SLALOM demonstrates the capacity to deliver a range of insightful explanations with both synthetic and real-world datasets. We highlight SLALOM's unique efficinncy-quality curve by showing that SLALOM can produce explanations with substantially higher fidelity than competing surrogate models or provide explanations of comparable quality a fraction of their computational costs. We release code for SLALOM an open-source project online at http:://gittub.comm/eeemam swwllmmllmmmmmmmmmmam haaaamammnlllllooo.peppaatttttot
https://githut: com
1
Introduction
The transformer architecture (Vaswani 2017) has been established
the status quo in modern natural Touvron al.) 2023). However, the current and foreseeable adoption of large language models (LLMs) in critical domains such the judicial system (Chalkidis et al.] 2019) and the medical domain (Jeblick et al.
language processing (Devlin al.] 2018 Radford al.) 2018 2019
2023) comes with an increased need
for transparency and interpretability. Methods to enhance the interpretability of an artificial intelligence
(AI) system are developed in the research area of Explainable AI (XAI, Adadi & Berrada 2018 Gilpin
et al. 2018 Molnar 2019: Burkart & Huber 202T). recent meta-study (Rong al.) 2023) shows that XAI has the potential to increase users' understanding AI systems and their trust therein. Local feature altribution methods that quantify the contribution of each input decision outcome are among the most. popular explanation methods, and a variety of approaches have been suggested for the task of computing such attributions [Kasncci & Gottron 2016 Ribeiro et al. 2016 Sundararajan et al. 2017 Lundberg & Lec
2017
Covert et al. 2021 Modarressi et al. 2022
It remains hard to formally define the contribution of an input feature for non-linear functions. Recent work Han al.) 2022) has shown that common explanation methods do so by implicitly or explicitly performing a local approximation of the complex black-box function, denoted as f, using a simpler surrogate function g from predefined class G. For instance, Local Interpretable Model-agnostic Explanations (LIME, Ribeiro

---

Published in Transactions on Machine Learning Research (12/2024)
Input sequence t [positive: o0]5,
[(]:
this
is
Faniastic
movie -
BERT
1.09
linear
0.93
Common Surrogate Models in XAI
Add prefix и (neutral: [-0.5, 0.5)):
f(u):
Predictive
Surrogate
Explanation Techniques
booo
Long
sinoe
Ssow
the
Models
Models
movic
becousee
something
Jalways
[happened
:o
BERT
0.29
Logistic / Linear
Linesr Model:
linear
0.19
Regression,
Kao non-linearities
C-LIME, Gradients, IG
Icome
up]
hooeever
f(t) f(a):
ReLU networks
Kno interactions
Removal-based, n.g. Shapley
Concatenation
[e, ul::
2 BERT 1.38
GAM, ensemblrs,
GAM: non--inearities Kno interactions
linsar
1.12
boosting
Values, Local GAM approxination
Ias
boca
long
Linse
stace
MC
Saw
the
Bast
f(tta)): vl):
SLALOM (ours):
bocause
somehhing
allaays
happened
come
BERT
0.45
Tranformers
/non-linearinn linearities
Local SLALOM approximation
however
this
Fantastic
movic
linear
1.12
/Iuteractions
Figure 1: Transformers cannot be well explained through additive models. Left: We exemplarily show the log odds for the outputs of a BERT model and linear Naive-Bayes model ("linear") assigning each word a weight trained on the IMDB movie review dataset. The token colors indicate the weights assigned by the linear model. We pass two sequences to the models independently and in concatenation. For the linear model, the output of the concatenated sequence can be described by the sum, but this not the case for BERT. We show that this phenomenon is not due to a non-linearity in this particular model but stems from general incapacity of transformers to represent additive functions. Right: To overcome this difficulty, we propose SLALOM, a novel surrogate model specifically designed to better approximate transformer models.
et al. 2016) imput gradient explanations (Baehrens et al. 2010 use a linear surrogate model to approximate the black-box f; the model's coefficients can be used as the feature contributions.
Surrogate model explanations have the advantage that they directly describe the behavior of the model in the proximity of a specific input, i.c., under small perturbations, a property known as fidelity (Guidotti et al. 2018 Nauta et al. 2023). Fidelity can be quantified through the difference between the prediction model's outputs and the surrogate model's outputs (Yeh et al., 2019 Zhou et al. 2019). Explanations that quantitatively describe the prediction model's output under perturbations with low error have high fidelity.
An implication of models with high representative capacity and high-fidelity explanations is the recovery property: If the true relation f between features and labels in the data is already within the function class G, the model will learn this function and we can effectively reconstruct the original f from the explanations. For example, suppose that the black-box function we consider is of lincar form, i.c., f(x) = -ww" T x, and has been correctly learned. In this case, a gradient explanation as well as continuous LIME (C-LIME, Agarwal et al.] (2021) will recover the original model's parameters up to an offset (Han al.. 2022 Theorem 1). Shapley value explanations {Lundberg & Lee. 2017) possess comparable relationship: It is known that they correspond to the pointwise feature contributions of Generalized Additive Models (GAM, Bordt & von Luxburg, 2023). The significance of recovery properties lies in their role when explanations are leveraged to gain insights into the underlying data. Particularly when XAI is used for scientific applications such as drug discovery (Mak et al., 2023), preserving the path from the input data to the explanation through a learned model is crucial. However, such guarantees can only be provided when surrogate function class G can effectively mimic the model's 's learned relation, at least within some local region.
In this study, we demonstrate that the transformer architecture, the main building block of LLMs such the GPT models (Radford et all 2019), is inherently incapable of learning additive models on the input tokens, both theoretically and empirically. By additíve models, we refer to models that assign each token a weight. The sum of the individual token weights then gives the output of the additive model. Linear models are a subset of this class. We formally prove that simple encoder-only and decoder-only transformers structurally cannot represent such additive models due to the attention mechanism's softmax normalization, which necessarily introduces token dependencies over the entire sequence length. An example is illustrated in Figure | (left). Our finding that the function spaces represented by additive models and transformers are
2

---

Published in Transactions on Machine Learning Research (12/2024)
disjoint when dismissing trivial cases implies that prevalent feature attribution explanations insufficient to model transformers. They cannot possess high fidelity, i.e., they cannot, quantitatively describe these models' behavior well. This also undermines the recovery property, highlighting a significant oversight in current XAI practices. As our results suggest that the rolc of tokens cannot be described through a single score, we introduce the Softmax-Linked Additive Log Odds Model (SLALOM, cf. Figure | right), which represents the role of cach input token in two dimensions: The token value describes the independent effect of token, whereas the token imporlance provides token's interaction weight when combined with other tokens in sequence. In summary, our work offers the following contributions beyond the related literature:
(1) We theoretically and empirically demonstrate that common transformer architectures fail to represent additive and linear models the input tokens, jeopardizing current attribution methods' fidelity, Additive Log Odds Model (SLALOM),
(2) To mitigate these issues, we propose the Softmax-Linie which uses a combination of two scores to quantify the role of input lokens.
(3) We theoretically analyze SLALOM and show that (i) it can be represented by transformers (i.e., the fidelity property), (ii) it can be uniquely identified from data (i.e., the recovery property), and (iii) it is highly efficient to estimate.
(4) Experiments on synthetic and real-world datasets with common language models (LMs) confirm the mismatch between surrogate models and predictive models, underline that two scores cover different angles of interpretability, and that SLALOM explanations can be computed that have substantially higher fidelity or elficiency than competing techniques.
2 Related Work
Explainability for transformers. Various methods exist to tackle model explainability (Molnar 2019
Burkart& & Huber 2021). Furthermore, specific approaches have been devised for the transformer architecture VVaswani et 2017): As the attention mechanism at the heart of transformer models is supposed to focus relevant tokens, it seems a good target for explainability methods. Several works turn to attention patterns as model explanation techniques. A central attention-based method is put forward by Abnar & Zuidema (2020). who propose two methods of aggregating raw attentions across layers, flow and rollout. Brunner et al. (2020) focus on effective attentions, which aim to identify the portion of attention weights actually influencing the model's decision. While these approaches follow a scalar approach considering only attention weights, Kobayashi et al. 2020 2023) propose a norm-based vector-valued analysis, arguing that relying solely on attention weights is insufficient and the other components of the model need to be considered. Building the norm-based approach, MModaressi et al. (2022 2023) further follow down the path of decomposing the transformer architecture, presenting global-level explanations with the help of rollout. Beyond that, many more attention-based explanation approaches have been put forward (Chen et al., 2020 Hao et al. 2021
Ferrando & Costa- jussá 2021 Qiang el al. 2022 Sun et al. 2023 Yang el al. |023) and
Lion methods such as TRP have been adapted to the transformer architecture (Achtibat et al., 2024). A drawback with these model-specific explanations remains the implementational overhead that is required to adapt these methods for each architecture. On the formal side, there is no explicit method to quantitatively predict the transformer model's 's behavior under perturbations leaving the fidelity of these explanations unclear.
Model-agnostic XAL. In contrast to transsormer--peeeffe methods, researchers have devised model-agnostic explanations that can be applied without precise knowledge of a model's architecture. Model-agnostic local explanations like LIME (Ribeiro et al., 2016), SHAP (Lundberg & Lee 2017] and others (Shrikumar ct al. 2017 Sundararajan et al., |017] /Smikovv et al. 2017 Xu et al.! 2020) Covert et al. 2021 etc.) are a
particularly popular class of explanations that are applied to LMs as well (Saczepański et al. 2021 Schirmer
et al. 2023 Dolk et al.) 2022 Surrogate models are common subform (Han et al. 2022 which locally
approximate a black-box model through a simple, interpretable function.
Linking models and explanations. Prior work has distilled the link between classes of surrogate models
that can be recovered by explanations (Agarwal et al.] 2021 Han et al. 2022, Theorem 3). Notable works
include Garreau & von Luxburg (2020), which provides analytical results on different paramotrizations of
LIME, and Bordt & von Luxburg (2023), which formalizes the connection between Shapley values and GAMs for classical Shapley values as well as n-Shapley values that can also model higher-order interactions.
3

---

Published in Transactions on Machine Learning Research (12/2024)
Interpreting feature attributions as linear models. Many feature attribution methods do not use explicit surrogate models, e-g., SHAP or LRP. To predict model behavior under perturbations they can be intuitively interpreted as a linear surrggate model (cf. Yeh et al.) 2019): If feature i has a contribution of di removing should reduce the model output by Φs (Achtibat et al. 2024), giving rise to an implicit lincar model.
We contribute the literature by theoretically showing that transformers are inherently incapable to represent additive models, casting doubts on the applicability of local LIME, SHAP, attention-based. and other attribution methods to transformers. These methods fit explicit additive surrogate model can be interpreted as onc. Our finding that additive scores are insufficient to predict the behavior of transformers leaves us with no method that enables strict fidelity. To bridge this gap, we provide SLALOM, novel surrogate model with substantially increased fidelity and a recovery property for transformers.
3 Preliminaries
logit score
AP
3.1 Input and output representations
cls
xL
In this work, we focus on classification problems of token sequences. For the sake of simplicity, we initially consider 2-class classification problem with labels y E y {0, 1}. We will outline how to generalize approach to multi-class problems in Appendix C.1
ffi:
fil
fln
P
A
8
h
(1-1)
[
The input consists of a sequence of tokens t- [4...... 4e] where |t] E I.....C C is the sequence length that can span at most C' tokens (the context length). All tokens t, in the sequence stem from a finite size vocabulary V, i.e., & E V. Vi = I...., |t]. To transform the tokens into a repre- sentation amenable to processing with computational meth- ods, the tokens need to be encoded as numerical vectors. To this end, an embedding function e V + Rdd is used, where d is the embedding dimension. Let e, = e(t.) be the embedding of the i-th token. The output is given by a logit vector E "I such that softmax(I) contains individual class probabilities.
multi-hrud attention block
emmbedding ei
embeddingg e:
emnbedding e
Figure 2: Transformer architecture. In each layer 1=1,... L, input embeddings h (1-1) for each token i are transformed into output embeddings h, When detaching the part prior to the classification head ("cls"), we see that the output only depends on the last em- bedding ht (?(--) and attention output $1.
3.2 The common transformer architecture
Many popular LMs follow the transformer architecture introduced by Vaswani et al. (011)) with only minor modifications. We will introduce the most relevant building blocks of the architecture in this section. A complete formalization is given in Appendix B.1 A schematic overview of the architecture is visualized in Figure 2 Let us denote the input embedding of token i i= I,..., |t| in layer / E I,..., L by h ERd
where (0) =e-- The core component of the attention architecture is the attention head For cach token, a query, key, and value vector are computed by applying an affine-linear transform to the input embeddings. Keys and queries are projected onto each other and normalized by a row-wise softmax operation resulting in attention weights Oaj [0, 1], denoting how much token is influenced by token j. The attention output for token i can be computed as = E/e "sy" where by E Rdd denotes the value vector for token j. The final 8; are projected back to dimension d by a projection operator P Rd. Rd before they are added to the corresponding input embedding (1(11)) as mandated by skip-connections. The sum is then transformed by a nonlinear function that denote by fin : Rd + Rd, finally resulting in transformed embedding h."). This procedure is repeated iteratively for layers 1,..., L such that we finally arrive at output embeddings h (L) To perform classification, a classification head cls : Rd + R is put on top of a token at some index (how
Although we only formalize single head here, our theoretical results cover multiple heads well
4

---

Published in Transactions on Machine Learning Research (12/2024)
this token is chosen depends the architecture, common choices include E {1, [t),), such that get the final logit output 1 = cls (2 The logit output is transformed to a probability vector via another softmax operation. Note that in the two-class case, we obtain the log odds F(t) by taking the difference (A) between the two logits, ie., F(t) = log gi==100 p(y=0
(() 1 lo-
3.3 Encoder-only and decoder-only models
Practical architectures can be seen as parametrizations the process described previously. We introduce the ones relevant to this work in this section. Commonly, distinction is made between encoder-only models, that include BERT (Devlin al. 2018) and its variants, and decoder-only models such as the GPT models Radford et al. 2018 2019).
Encoder-only models. Considering BERT as an example of an encoder-only model, the first token is used for the classification head, i.e., r = 1. Usually, a special token [CLS] is prepended to the text at position 1. However this is not strictly necessary for the functioning of the model.
Decoder-only models. In contrast, decoder-only models like GPT-2 (Radford et al., 2019) add the classification head on top of the last token for classification, i.e., r r= |t] A key difference is that in GPT-2 and other decodcr-only models, a causal mask is laid over the attention matrix, resulting in Oi.j 0 for i. This encodes the constraint that tokens can only attend to themselves to previous ones.
To make our model amenable to theoretical analysis, the transformer model in our analysis contains one slight deviation from practical models. We do not consider positional embeddings, which are added to the embeddings based on their position the sentence. We will empirically demonstrate that using positional embeddings does not affect the validity of our findings on practical models.
4 Analysis
Let us initially consider a transformer with only single layer and head. Our first insight is that the classification output can be determined only by two values: the input embedding at the classification token r, h and the attention output 8,- This can be seen when plugging in the different steps:
a(t)=A(cls( (cls(h!")) = A (cls (ffn(h)))) + P(s.)))) = g(, 8,) = 9 =((n
Σ
arj
(1)
f=1
The attention output is given by a sum of the token value vectors vj weighted by the respective attention weights arj-
4.1 Transformers cannot represent additive models
We now consider how this architecture would represent linear model. In this model, each token is assigned weight w : V > IR. The output is obtained by adding weights and offset E IR, consequently requiring
F( .., - b Σ ((.)
for all possible input sequences.
The transformer shows surprising behavior when considering sequences of identical tokens but of different. lengths, i.e., [T,] We first note that the sum of the attention scores is bound to be Σ/= arj 1. The output of the attention head will thus be a weighted average of the value vectors U,. We find that the first-layer value vectors (t,))) are determined purely by the input tokens tý (cf. full formalization in Appendix B.I. For a sequence of identical lokens, we will thus have the same value vectors, resulting in identical vectors being averaged. This makes the transformer produce the same output for each of these soquences. This contradicts the form in (2), where the output should successively incrcase by w(t,). We are now ready to state our result, which formalizes this intuition for the more general class of additive models where the token weight may also depend on its position i in the input (equivalent to GAM).
5

---

Published in Transactions on Machine Learning Research (12/2024)
Proposition 4.1 (Syrgle-layer transformers cannot represent additive models.). Let V be vocabulary and C2 2, be mazimum sequence length (contert length). Tel wy IR, Vi 1,.... C' be any map that assigns token encountered at position i numerical score including at least one token T E V with non-zero weight Wx(T) + 0 for some i C.....C. Let b E R be an arbitrary offset. Then, there crists no parametrization of the encoder or decoder single-layer transformer F such that for every sequence t = [1,, 12, (2......4 M/j with length C, the oulput of the transformer nelwork is equivalent lo F(11,122-... (11l)) b+ E = w,(4))
Proof Sketch. We prove the statement by concatenating the token T to sequences of different length. We then show that the inputs to the final part of the transformer will be independent of the sequence length. Due to being deterministic, the output will also be independent the sequence length. This contradictory to the additive model with a weight (ry(r) o requiring different. outputs for sequences of length j j-1 and j. Formal proofs for all results can be found in Appendix
]
In simple terms, the proposition states that the transformer cannot represent any additive models on sequences of more than one token besides constant functions or those fully determined by the first input token. Importantly, the class of functions stated in the above theorem includes the prominent case of linear models in Eqn. [ where each token has a certain weight w independent of its position in the input vector (i.e., Wi w. Vi, see Corollary B.2). We would like to emphasize that this statement includes the converse:
Corollary 4.2. Transformers whose oulputs are not constant or fully determined by the first token of the input sequence cannot be functionally equivalent to an additive model.
4.2 Transformer networks with multiple layers cannot represent additive models
In this section, we will show how the argument can be extended to multi-layer transformer networks, Denote by h(-1)) the input embedding of the ith token at the Ith layer. The output is governed by the recursive relation
h (h + Pi())) - (h (1)) s:). ss)).
Exploiting the similar form allows us to generalize the main results to more layers recursively.
Corollary 4.3 (Multi-Layer transformers cannot learn additive models either). Under the same conditions as in Proposition 4.1, a stack of multiple transformer blocks as the model F, neither has a parametrization sufficient to represent the additive model.
Practical considerations. As stated earlier, the transformer model in our analysis does not consider positional embeddings that are added on the token embeddings. However, this does not have major ramifications in practice: While the transformer would be able differentiate between sequences of different lengths with positional embeddings in theory, the softmax operation must be inverted for any input sequence by the linear feed-forward block that follows the attention mechanism. This is highly non-linear operation and the number of possible sequences grows exponentially with the context length and vocabulary size. -earning-theoretic theoretic considerations suggest that this inversion is impossible for reasonably-sized networks as outlined in Appendix C.2 We will confirm our results with empirical findings obtained exclusively on non-modified models with positional embeddings.
5 A Surrogate Model for Transformers
In the previous section, we thcoretically established that transformer models struggle to represent additive functions. While this must not necessarily be considered a weakness, it certainly casts doubts on the suitability of additive models as surrogate models for explanations of transformers. For a principled approach, we consider the following four requirements to be of importance:
(1) Interpretability. The surrogate model should be simple enough such that its parameters are inherently interpretable for humans (Molnar 2019 Chapter 9.2).

---

Published in Transactions on Machine Learning Research (12/2024)
16
layersa
layenez
10
10
layonent
10
10
layencc
uucrelation
m400
1000
10
15
20
10
1.5
20
10
15
10
15
"perfeet")
choce. "perteet")
(Wocc. "perloet")
(fooc.. "perfect")
(a) Fully-Connedee Model
(b) BERT
(c) DistiLBERT
(d) GPTT-2
Figure 3: Transformers fail to learn linear models. We train different models on synthetically sampled dataset where the log odds obey linear relation to the features. Fully connected models (2-layer ReLU networks with different hidden layer widths) capture the linear form of the relationship well despite some estimation error (a). However, common transformer models fail to model this relationship and output almost constant values (b)-(d). This does not change with more layers.
(2) Learnability. The surrogate model should be easily representable by common transformers. Learn- ability crucial because using a surrogate model that is hard to represent for the predictive model will likely result in low-fidelity explanations.
(3) Recovery. If the predictive model falls into the surrogate model's class, the fitted surrogate model's parameters should match those of the predictive model. Together with learnability, recovery ensures that model can pick up the true relations present in the data and they can be re-identified by the explanation, which is essential, e.g., in scientific discovery with XAI.
(4) Efficiency. The surrogate model should be efficient to estimate even for larger models.
5.1 The Softmax-Linked Additive Log Odds Model
To meet the requirements, we propose novel discriminative surrogate model that explicitly models the behavior of the softmax function. Instead of only assigning a single weight w to each token, we separate two characteristics: We introduce the token importance as a map s: V + R and a token value form of a map v + R. Subsequently, we consider the following discriminative model:
F(t) = log ply 11t) ply 0lt)
where a;(t) = exp(s(14))
Σ (t(t))(ts), TEEE
(4)
E, exp()(l,))
Due the thift-invvriance of the softmax function, we observe that the maps and s' given by s'(T) s(T) (())+ result in the same softmax-score and thus the same log odds model for any input t. Therefore, the parameterization would not be unique. To this end, we introduce normalization constraint on the sum of token importances for uniqueness. Formally, we constrain it to a user-defined constant Y E RR such that Erey (()) = 7, where natural choices include 7 E {0, 1}. We refer to the discriminative model given Eqn. (4) together with the normalization constraint as the softmaz-linked additive log odds model (SLALOM).
As common in surrogate model explanations, we fit SLALOM a predictive model's outputs globally locally and use tuples of token importance scores and token values scores, (u(T), s(T)) to give explanations for an input token т. While the value score provides an absolute contribution of to the output, its token importance s()) determines its weight with respect to the other tokens. For instance, if only one token 7 is present in a sequence, the output is only determined by its value score 6(7). However, a sequence of multiple tokens, the importance of each token with respect to the others and thereby the contribution of this token's 's value- is determined by the token importance scores This intuitive relation makes SLALOM interpretable, thereby satisfying Property (1).
7

---

Published in Transactions on Machine Learning Research (12/2024)
5.2 Theoretical properties of SLALOM
We analyze the proposed SLALOM theoretically to ensure that it fulfills Properties (2) and (3), Learnability and Recovery, and subsequently provide efficient algorithms to estimate its parameters (4). First, we show that unlike linear models, SLALOMs can be easily learned by transformers.
Proposition 5.1 (Transformers can fit SLALOM). For any maps 8, v, and a transformer with an embedding size d and head dimension dn. with d, dn 2 3, there exists a paramelerizalion, of the transformer to reflect SLALOM in Equation (4) together with the normalization constraint.
This statement can be proven by explicitly constructing the corresponding weight matrix (cf. Appendix B.5 This proposition highlights that - unlike linear models - there are simple ways for the transformer to represent relations governed by SLALOMs. We demonstrate this empirically in our experimental section and conclude that SLALOM fulfills Property (2). For Property (3), Recovery, we make the following proposition:
Proposition 5.2 (Recovery of SLALOMs). Suppose query access to a model G that takes sequences of tokens
with lengths |t] E 1,....C. C and returns the log odds according to a non-constant. SLALOM on a vocabulary
V, normalization constant T E, but with unknown parameter mops s : + R, U R. For C' 2 2, we can recover the true maps 8, with 2 - forward passes of F.
This statement. confirms property (3) and shows that SLALOM can be uniquely re-identified when we rule out the corner case of constant models. We prove it in Appendix B.6
Complexity considerations. Computational complexity can be concern for XAI methods. To estimate exact. Shapley values, the model's output on exponentially many feature coalitions needs to be evaluated. However, as the proof of Proposition 5.2 shows, to estimate SLALOM's parameters for an input sequence of V tokens, only 2| forward passes are required, verifying Property (4). We empirically show that computing SLALOM explanations is about 5x faster than computing SHAP explanations when using the same number of samples in experimental section.
5.3 Numerical algorithms for computing SLALOMs
Having derived SLALOM as belter surrogate model, we require numerical algorithms to estimate v and Unfortunately, the strategy derived in Proposition 5.2 using a minimal number of samples is numerically unstable. We make two key implementation choices for SLALOM to be used as an explanation technique. First, we can control the sample set of features and labels obtained through queries of the predictive model. Second, we can use different optimization strategies to fit. SLAL.OM on this sample set. We suggest two algorithms to fit SLALOMs post-hoc on input-output pairs of a trained predictive model:
SLALOM-eff.
The first version of the algorithm to fit SLALOM models is designed for maximum efficiency while maintaining reasonable performance across several XAI metrics. Obtaining large dataset of input- output pairs can incur substantial computational costs as a forward pass of the models needs to be executed for each sample. To speed up this process, SLALOM-EFf uses very short sequences (we use only two tokens in this work) randomly sampled from the vocabulary for this purpose. To efficiently fit the surrogate model, we perform stochastic gradient descent on SLALOM's paramcters using the mean-squared-error loss between the score output by SLALOM and the score by the predictive model as an objective. SLALOM-eff is our default technique used unless stated otherwise.
We provide another technique to fit SLALOM optimized for maximum fidelity under input perturbations such as loken removals. To explain a specific sample, we sample input where we remove up to K randomly sampled tokens. The sequences with tokens removed and their predictive model scores are used to fit the model, similar to LIME (Ribciro et al. 2016). Instead of SGD, we can leverage optimizers for Least-Square- Proparr-roobless to fit the parameters iteratively, however incurring a higher latency. We provide details and pseudocode for both fitting routines in Appendix
SLALOM-fidel.
8

---

Published in Transactions on Machine Learning Research (12/2024)
log odds (fitted disttibeert)
GAM Fend
GAM
immort scorr
10
recovered
original
Jog odds
log-nals
[V] = 128
-2
RMSE-0.003
2.5
3 (Booz of" "pallstt
[Ausc, of "petfost")
log odds, ground truth
value score
(a) Fitting SLALOM to (b) Filting SLALOM to (c) Comparing SLALOM and (d) Recovering parameters of SLALOM dalaset
BERT outpuls
GPT-2 outputs
predictive output
Figure 4: Verifying properties with syuthetic data: SLALOM describes outputs of transformer models well (a, b). We fit SLALOM to the outputs of the BERT and GPT-2 models trained on the linear synthetic dataset. The linear and GAM models (despite having C/2-15x more parameters) do not match the transformer's s behavior. We provide another empirical counterexample and additional quantitative results in Appendix F.1 Verifying recovery dc. We verify the recovery property on second synthetic dataset where features and labels obey a SLALOM relation. We train a 2-layer DistilBERT model on the data and SLALOM to the trained model. We can recover the original logit scores (c) and see a strong connection between original SLALOM parameters and the recovered ones (d). These findings verify the learnability and recovery properties. More results in Appendix F.2
5.4 Relating SLALOM scores to linear attributions
Importantly, SLALOM scores can be readily converted to locally linear interpretability scores where necessary. For this purpose, differentiable model for soft removals is required. We consider the weighted model:
F(A) Enst d exp(s(()))((t)
(5)
Ener d exp(s((4))
where A = 1 if a effect of making the output of the soft-removal model equivalent to that the standard SLALOM on a sequence without this token. Taking the gradients al À 1 we obtain OF ax: x=1
a token is present and A; = 0 if it is absent. We observe that setting A; = 0 has the desired
x(4) ex4))x)))) which can be used to rank tokens according to the linearized attributions. We defer the derivation to Appendix B.7 and refer to these scores as linearized SLALLOM scores. As SLALLM is just another multi-variate function, it also possible to compute SHAP values for it. We discuss this relation in Appendix B.7 as well.
6 Experimental Evaluation
We run series of experiments to show the mismatch between surrogate model explanations and the transformers. Specifically, we verify that (1) real transformers fail to learn additive models, (2) SLALOM better captures transformer output, (3) SLALOM models can be recovered from fitted models with tolerable error, (4) SLALOM scores are versatile and align well with linear attribution scores and human attention, and (5) that SLALOM performs well in faithfulness metrics and has substantially higher fidelity than competing techniques. For experiments (1)-(3), we require knowledge of the ground truth and use synthetic datascts. To demonstrate the practical strengths of our method, all the experiments for (4) and (5) are conducted on real-world datasets and in comparison with state-of-the-arl XAI Lechniques.
6.1 Experimental setup
LM architectures. We study three representative transformer language model architectures in our experiments. In sequence classification, mid-sized transformer LMs are most popular on platforms such as the Huggingface hub (Huggingface 2023) often based on the is reflected in experimental setup. To represent the family of encoder-only models, deploy BERT
(Devlin al.) (0088), which
9

---

Published in Transactions on Machine Learning Research (12/2024)
Devlin et al. 2018 and DistilBERT (Sanh et al. 2019 We further experiment with GPT-2 RRdffrrd et al. 2019), which is decoder-only model. We use the transformers framework to run our experiments. While not the main scope of this work concerned with sequence classification, we will also show that due to its model-agnostic nature, SLALOM can be applied to LLMs with up to 7B parameters and non-transformer models such as Mamba (Gu & Dao, 2023) with plausible results. We provide a proof-of-concepl to show that, SLALOM can be applied to black-box models such as GPT-4o Appendix
F.7
Datasets. We use synthetic datasets and two real-world datasets for sentiment classification. Specifically, study the IMDB dataset, consisting of movie reviews (Maas et al., 2011) and Yelp-HAT, dataset of restaurant reviews, for which human annotators have provided annotations on which tokens are relevant for the classification outcome (Sen et al. 2020). We provide additional details on hyperparameters, training, and datasets in Appendix E
6.2 Evaluation with known ground truth
Transformers fail to capture linear relationships. We empirically verify the claims made in Proposi- tion 4.1 and Corollary 4.3] To ascertain that the underlying relation captured by the models is additive, we resort to a synthetic dataset with a linear relation. The dataset is created as follows: First, we sample different sequence lengths from a binomial distribution with mcan of 15. Second, we sample words independently from vocabulary of size 10. This vocabulary was chosen to include positive words, negalive words, and neutral words, with manually assigned weights E {-1.5, -1, 0, 1, 1,5), that can be used to compute linear log odds model. We evaluate this model and finally sample the sequence label accordingly, thereby ensuring a linear relation between imput sequences and log odds. We train transformer models on this dataset and evaluate them on sequences containing the same word ("perfect") multiple times. Our results in Figure 3 show that the models fail to capture the relationship regardless of the model or number of layers used. In Appendix A we show how this undermines the recovery property with Shapley value explanations.
Fitting SLALOM as surrogate to transformer models. Having demonstrated the mismatch between additive functions and transformers, we turn to SLALOM as more suitable surrogate model. As shown in Proposition 5.1 transformers can easily fit SLALOMs, which is why we hypothesize that they should model the output of such a model well in practice. We fit the different surrogate models on a dataset of input sequences and real transformer outputs from our lincar synthotic dataset and observe that lincar models and additive models fail to capture the relationship learned by the transformer as shown in Figure 4 a b). On the contrary, SALO manages to model the relationship well, even if it has considerably less trainable parameters than the additive model (GAM).
Verifying recovery. We run an experiment to study whether, unlike linear models, SLALOM can be fitted and recovered by transformers. To test this, we sample a second synthetic dataset that exactly follows the relation given by SLALOM. We then train transformer models on this dataset. The results in Figure I d) show that the surrogate model fitted on transformer outputs as a post-hoc explanation recovers the correct log odds mandated by SLALOM (c) and that there is a good correspondence between the true model's parameters and the recovered model's parameters (d).
6.3 Examining real-world predictions from different angles
We increase the difficulty and deploy SLALOM (fitted using SLATOM-eff) to explain predictions on real- world datasots. As there is no ground truth for these datasots, it is challenging to evaluate the quality of the explanations (Rong al. 2022). To better understand SLALOM explanations, we study them from several angles: We compare to linear scores obtained when fitting Naive-Bayes Bag-of Words (BoW) model, scores on removal and insertion benchmarks (Tomsett et al. 2020 Younung et al. 2020), the human attention scores available on the Yelp-HAT dataset (Sen et al. 2020). and provide qualitative results.
Explaining Sentiment Classification. We show qualitative results for explaining a movie review in Figure 5 The figure shows that both negative and positive words are assigned high importance scores but have value scores of different signs. Furthermore, we see that some words ("the") have positive value scores, but very low importance. This means that they lead to positive scores their own but are easily
10

---

Published in Transactions on Machine Learning Research (12/2024)
LM
values e
imporcances a lin.
Bert 0.786 0.01
0.807 0.01 0,801 = 0.01
LM values e
importances
lin.
DistilBERT 0.688 + 0.01
0.681 = 0.01
0.686 + 0.01
BERT 0.619 +: 0.01 0.349 0.01
0.626 0.01
GPT 2 0.674 0.01
0.685
0.01 0.683 4 0.01
DistiBERT 0,692 +:2 0.01
0.373
0.01
0.693
0.01
GPT-20.618 + 0.01 0.292
0.01
0.619
0.01
average
0.716
0.724
0.724
average
0.643
0.338
0.646
BLOOM-77.B 0.739 0.02 0.712 + 0.03 0.740 0.02
(a) Measuring average rank-correlation (Spear- man) between Naive-Bayes scores and SLALOM scores. Linearized performs best.
Mamba- 2.8B 0.615 0.03 0.437 a 0.03 0.535 4 0.03
(b) Measuring average AU-ROC between SLALOM expla- nations and human token attention. The importance scores and most strongly predictive of human attention for the classical models. Applying SLALOM to larger models un- derlines its scalability, but it remains less reliable for non- transformer models like Mamba.
SLALOM-fidel
SLA
LM
t-scores
lin.
P-scores
lin.
LIME
SHAP
Grad
LRP
BERT
0.025 + 0.002
0.023 = 0.001
0.031 + 0.002
0.031 = 0.002
0.024 = 0.002
0.026 + 0.003
0.537
0.034
0.611 = 0.033
0.030 0.006
DistiBERT
0,028 + 0.003
0.024
0.002
0.027 0.002
0.027
0.002
0.027 + 0.003
0.029
0:003
0.495
0.027
0.508
0.028
0.023 4 0.002
GPT-2 0.052 + 0.008
0.050
+ 0.008
0.089 + 0.008
0.089 + 0.008
0:230 + 0.017
0.042 + 0.005
0.454 + 0.022
0.493 = 0.023
0.069 + 0.010
averago 0.035 + 0.004
0.032 = 0.004
0.049 + 0.004 0.049 + 0.004
0.004 + 0.007 0.032 + 0.003 0.502 + 0.028 0.537 + 0.028 0.041 + 0.006
(c) Area Over Perturbation Curve for deletion. Linearized scores and SHAP performs best in the XAI metric.
Table 1: Evaluation of SLALOM scores ("values", "importance", "lin.") with std. errors across explanation quality measures highlights that SLALOM's different scores serve differrnt purposes. IMDB dataset shown.
overruled by other words. We compare the SLALOM scores obtained on 100 random test samples to a linear Naïve- Bayes model (obtained though counting class-wise word frequencics) as surrogate ground truth in Table Ta through the Spearman rank correlation. We observe good agreement with the value scores v and the combined linearized SLALOM scores ("lin", see Section 5.4..
Predicting Human Attention. To study alignment with a user perspective, we predict human attention from SLALOM scores. We compute AU-ROC for predicting annotated human attention as suggested in Sen et al. (2020) in Table 1b. We use absolute values of all signed explanations as human attention is unsigned as well. In contrast to the previous experiments, where value scores were more effective than importances, we observe that the importance scores are often best at predicting where the human attention is placed. In summary, these findings highlight that the two scores serve different purposes and cover different dimensions of interpretability. SLALOM offers higher flexibility through its 2-dimensional representation.
We also use this opportunity to study the applicability of SLALOM to larger and non-transformer models. To this end, we train Mamba and BLOOM models (Le Scao et al. 2023) with a classification head on the
2.0
2.0 pad
bol
2.0
Jin
1.55
impontee scom
1.0
Foc this
We need
to
have
talk
That
littlo
girl
who
was
always
1.0
family
Jesutiful
hanging
someone
just
hateed
her
and
had
mentbon
0.5
0.5
Anerzetic
grbll Spoed
Now
the rinal
soone
transe
ends
must
for
iously
had and
Full of
bad
Iesss]
thot
is
a|
movie
of
its
ooen
What
-05
crappy
dancing
Hor
ribll
and
beautiful
oncee
value score
Value S000t
(a) BERT
(b) GPT-2
(c) Token highlighting (linearized SLALOM, GPT-2)
Figure 5: Explaining a real review with SLALOM (qualitative results). SLALOM assigns two scores to each token ba and can be used to compute attributions via its linearization (c). We observe that the impactful words have high importances and the value scores indicate the sign of their contribution (positive or negative words). See Figure 12 (Appendix) for fully annotated plots.
11

---

Published in Transactions on Machine Learning Research (12/2024)
10
10"
100
pralction
SLALOSTIAR RAET SLALOMEST
Gruz
prodection MSSI
30'
MSE
EIMR
IRP
10"
SHAP
10-1
10m4
455
41%
10
10
10
token mmmosis
6 token remonals
toker remecaal
BERT
(b) DistilBERT
(c) GPT-2
Figure 6: Asscssing Fidelity. We plot the MSE for predicting model outputs under token removal and find that SLALOM's predictions have up to 70% less error than the closest competitor when up to 10 random tokens from a sentence are removed (log-y plots). We interpret LRP scores as a linear model.
Yelp dataset. Our results in the lower part of Table Ib prove that SLALOM can be well applied to larger models with sizes of up to 7B. For BLOOM the results are promising and reach the same level as for the classical transformer models. SLALOM can also be applied to non-transformer models like Mamba (Gu & Dao 2023) due to its mgnet-agnostic nature. Due to its general expessivity, the explanations are capable of predicting human attention (at least the value scores), but we observe slight reduction in quality. We thus see SLALOM's main scope with models that follow the classical transformer paradigm. We show that STAALOM's results for BoW correlations and human attention prediction in the same range and often outperform competing XAI techniques in Appendix F.2.2 but defer comparative analysis to the next sections.
Assessing Fidelity. Having established the roles of its components, we verify that SLALOM can produce explanations that have substantially higher fidelity than competing surrogate or non-surrogate explanation techniques. To assess this, we remove up 10 tokens from the input sequences and use the explanations predict the change in model output using the surrogate model (SLALOM or linear). We compare the two SLALOM versions to baselines such as LIME (Ribeiro et al., 2016), Kernel-SHAP (Lundberg & Lee 2017), Gradients (Simonyan et al. 2013), and Integrated Gradients (IG, Sundararajan et al.) 2017) and layer-wise relevance propagation (LRP) for transformers (Achtibat et al. 2024), a non-surrogate technique. We report the Mean-Squared- Error (MSE) between the predicted change and the observed change when running the model on the modified imputs in Figure 0 We observe that SLALOM-fidel offers substantially higher fidelity with reduction of 70% in MSE for predicting the output changes over the second-best method (LRP).. Other surrogate approaches and LRP remain cluttered together, potentially highlighting the frontier of maximum fidelity possible with lincar surrogate model.
Evaluating XAI Metrics. There are several other metrics to quantify the quality of explanations and to compare different explanation tech- niques. As a sanity check and to show that SLALOM explanations do not lag behind other techniques in established metrics, we run the classical insertion/removt
Approach
Avg.
Time (s)
Grad
0.01 + 0.00
IG
0.02 + 0.00
LRP
0.02 = 0.00
benchmarks (Tomsett et al., 2020). For the insertion benchmark, we successively add the tokens with the highest attributions to the sample, which should result in high score for the target class. We iteratively insert more tokens and compute the "Area Over the Per- turbation Curve" (AOPC, see DeYoung et al. (2020)), which should be low for insertion. This metric quantifies the alignment of explanations and model behavior but only considers the feature ranking and not the assigned score. For surrogate techniques (LIME, SHAP, SLALOM) we use 5000 samples each. Our results in Tab. Ic highlight that lincarized SLALOM scores outperform LIME and LRP and perform on par with SHAP. Removal results for IMDB and results YELP with similar
SLALOM-eff
2.03 1 0.01
SLALOM-fidel
3.77 = 0.24
LIME
3.93
SHAP
11.56 + 0.03
Table 2: Runtime comparison us- ing 5000 samples to estimate sur- rogate models.
12

---

Published in Transactions on Machine Learning Research (12/2024)
findings are deferred to Table [1 (Appendix). In conclusion, this shows that top of SLALOM's desirable properties, it is on par with other techiques in common evaluation metrics.
Computational Costs. Finally, we take a look at the computational costs of the methods, which are mainly determined by sampling the dataset to fit the surrogate model. We provide the runtimes to explain a sample on our hardware when using 5000 samples estimate surrogates Table 2 with more results Appendix F.6. We observe that SHAP incurs the highest computational burden. Among surrogate model explanations SLALOM-eff is the most efficient, being about 5x more efficient that SHAP and 2x more efficient than LIME. Nevertheless, non-surrogate techniques are far more efficient as they require only few (IG steps) forward or backward passes, but suffer from other disadvantages (e.g., implementation effort, no explicit way to predict model behavior). Overall, our results highlight that the two SLALOM fitting routines can produce explanations of comparable utility to other surrogate models at a fraction of the costs, or produce explanations with higher fidelity at similar costs due to structurally better alignment between surrogate and predictive models.
7 Discussion and Conclusion
this work, we established that transformer networks are inherently incapable of representing linear or additive models commonly used for feature attribution. We prove that the function spaces learnable by transformers and linear models disjoint when ruling out trivial cases. This may explain similar incapacities observed in time-series forecasting (Zeng et al., 2023). where they seem incapable of representing certain relations. To address this shortcoming, we have introduced the Softmaa--Lined Additive Log Odds Model (SLALOM), a surrogate model for explaining the influence of features on transformers and other complex LMs through a two-dimensional representation.
Our work still has certain limitations that could be addressed in future work. SLALOM specifically designed explain the behavior of transformer models and therefore aligned with the classes of functions commonly represented by transformers. However, it would not be suitable choice to explain models capturing linear relationship. While SLALOM is generally applicable to any token-based LMs, we recommend using SLALOM only when the model is known to have attention-like non-linearities. Our results indicate that performance for these models is highest. We also note that SLALOM operates at the token level by assigning each individual token importance and value scores. Contextual or higher-order interpretability considering the meaning and impact of phrases, clauses, or sentences not covered by SLALLO.. To complement this theoretical foundation, future work will include further evaluation of SLALOM from a user-centric perspective, for instance, on human-centered evaluation frameworks (Colin et al., (022).. From a broader perspective, we hope that this research paves the way for advancing the interpretability and theoretical understanding of widely adopted transformer models.
Broader Impact Statement
This paper presents theoretical work on better understanding feature attributions in the transformer framework. We advise using caution when using our XAT technique or other model explanation as all explanations present only a simplified view the complex ML model. Our method works best with models of the transformer architecture. Besides that, we do not see any immodiate impact which we feel must be spocifically highlightod here.
References
Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4190 4197, 2020.
Reduan Achtibat, Sayed Mohammad Vakilzadch Hatefi, Maximilian Dreyer, Aakriti Jain, Thomas Wiegand, Sebastian Lapuschkin, and Wojeiech Sanck. Attnlrp: Attention- aware layer-wiss relevance propagation for transformers. Tn Forty-frrt International Conference Machine Learning, 2024.
13

---

Published in Transactions on Machine Learning Research (12/2024)
Amina Adadi and Mohammed Berrada. Pecking inside the black-box: survey explainable artificial intelligence (xai). IEEE access, 2018.
Sushant Agarwal, Shahin Jabbari, Chirag Agarwal, Sohini Upadhyay, Steven Wu, and Himabindu Lakkaraju. Towards the unification and robustness of perturbation and gradient based explanations. In International Conference on Machine Learning, pp. 110-119. PMLR, 2021.
Nabiha Asghar. Yelp dataset challenge: Review rating prediction. arxie preprint (rXiu:605500362, 2016.
David Bachrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klans-Robert Müller. How to explain individual classification decisions. The Journal of Machine Learning Research, 11: 1803 1831, 2010.
Peter Bartlett, Vitaly Maiorov, and Ron Meir. Almost linear ve dimension bounds for piecewise polynomial networks. Advances in neural information processing systems, 11, 1998.
Sebastian Bordt and Ulrike von Luxburg. From shapley values to generalized additive models and back. In International Conference on Artificial Intelligence and Statistics, pp. 709-745. PMLR, 2023.
Gino Brunner, Yang Liu, Damian Pascual, Oliver Richter, Massimiliano Ciaramita, and Roger Wattenhofer. On identifiability in transformers. 2020.
Nadia Burkart and Marco F Huber. A survey the explainability of supervised machine learning. Journal of Artificial Intelligence Research, 70:245-311 2021.
Javier Castro, Daniel Gómez, and Juan Tejada. Polynomial calculation of the shapley value based on sampling. Computers & Operations Research, 36(5):1726-1733 1730, 2009.
Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. Neural legal judgment prediction in english. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4317-430 4323, 2019.
Hanjie Chen, Guangtao Zheng, and Yangfeng Ji. Generating hierarchical explanations on text classification via feature interaction detection. arXie preprint arXiv:2004.00115 2020.
Julien Colin, Thomas Fel, Rémi Cadène, and Thomas Serre. What cannot predict, do not understand: A human-centered evaluation framework for explainability methods. Advances in neural information processing systems, 25:2832-2844 2022.
Ian Covert, Scott Lundberg, and Su-In Lec. Explaining by removing: A unified framework for model explanation. Journal of Machine Learning Research, 22(209):1 10, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. orXiv preprint 180=:080000400, 2018.
Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C Wallace. Eraser: A benchmark to evaluate rationalized nlp models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4443-4458, 2020.
Alexander Dolk, Hjalmar Davidsen, Hercules Dalianis, and Thomas Vakili. Evaluation of lime and shap in explaining automatic icd-10 classifications of swedish gastrointestinal discharge summaries. In Scandinavian Conference on Health Informatics, pp. 166 173, 2022.
Javier Ferrando and Marta. R Costa-jussà. Attention weights in transformer nmt fail aligning words between sequences but largely explain model predictions. arXiv preprint arXiv:2109.0588 05853, 2021.
Damien Garreau and Ulrike von Luxburg. Explaining the explainer: A first theoretical analysis of lime. In International conference on artificial intelligence and statistics, pp. 1287-1296. PMLR, 2020.
Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Explaining explanations: An overview of interpretability of machine learning. In DSAA, 2018.
14

---

Published in Transactions on Machine Learning Research (12/2024)
Albert Gu and Tri Dao. Mamba: Lincar-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.007522 2023.
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models. ACM computing surveys (CSUR), 51(5):1-42, 2018.
Tessa Han; Suraj Srinivas, and Himabindu Lakkaraju. Which explanation should choose? a function approximation perspective to characterizing post hoc explanations. Advances in Neural Information Processing Systems, 35:5256 5268, 2022.
Yaru Hao, Li Dong, Furn Wei, and Ke Xu. Self-attention attribution: Interpreting information interactions inside transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 12963 -2971,, 2021.
Huggingface. Hugging face - models: Most downloaded sequence classification models, 2023. URL https //huggingface
Katharina Jeblick, Balthasar Schachtner, Jakob Dexl, Andreas Mittermeier, Anna Theresa Stüber, Johanna Topalis, Tobias Weber, Philipp Wesp, Bastian Oliver Sabel, Jens Ricke, et al. Chatgpt makes medicine easy to swallow: an exploratory case study on simplified radiology reports. European radiology, pp. 1-9; 2023.
Gjergji Kasneci and Thomas Gottron. Licon: A linear weighting scheme for the contribution ofinput variables in deep artificial neural networks. In Proceedings of the 25th ACM international on conference on informalion and knowledge management, pp. 45 45-54, 2016.
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. Attention is not only weight: Analyzing transformers with vector norms. In 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, pp. 7057 7075. Association for Computational Linguistics (ACL), 2020.
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Tnui. Feod-forward blocks control contextual- ization in masked language models. arXiv preprint Xiie:2302.00456 2023.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. 2023.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142 150, 2011.
Kit-Kay Mak, Yi-Hang Wong, and Mallikarjuna Rao Pichika, Artificial intelligence in drug discovery and development. Drug Discovery and Evaluation: Safely and Pharmacokinetic Assays, pp. 1 38, 2023.
Sasan Maleki, Long Tran-Thanh, Greg Hines, Talal Rahwan, and Alex Rogers. Bounding the estimation error of sampling-based shapley value approximation. arXiv preprint arXiv:1306.4466 2013.
Ali Modarressi, Mohsen Fayyaz, Yadollah Yaghoobzadeh, and Mohammad Taher Pilehvar. Globenc: Quanti- fying global token attribution by incorporating the whole encoder layer in transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 258-271, 2022.
Ali Modarressi, Mohsen Fayyaz, Ehsan Aghazadeh, Yadollah Yaghoobzadeh, and Mohammad Taher Pilehvar. DecompX: Explaining transformers decisions by propagating token decomposition. In Proceedings of the 61sl Annual Meeling of the Associalion for Computational Linguistics ( (Volume 1: Long Popers), pp- 2023.
Christoph Molnar. Interpretable Machine Learning. 2019.
15

---

Published in Transactions on Machine Learning Research (12/2024)
Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, Jörg Schlötterer, Maurice Van Keulen, and Christin Seifert. From aneodotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai. ACM Computing Surveys, 55(13s):1 42, 2023.
Yao Qiang, Deng Pan, Chengyin Li, Xin Li, Rhongho Jang, and Dongxiao Zhu. Attcat: Explaining transformers via attentive class activation tokens. Adcances in Newral Information Processing Systems, 35: 5052-5064, 2022.
Alec Radford, Karthik Narasimhan, Tim Salimans, Nya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodci, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI I blog, 1(8):9, 2019.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144 2016.
Yao Rong, Tobias Leemann, Vadim Borisov, Gjergji Kasneci, and Enkelejda Kasneci. A consistent and efficient evaluation strategy for attribution methods. In International Conference Machine Learning, pp. 18770-18795. PMLR, 2022.
Yao Rong, Tobias Leemann, Thai-Trang Nguyen, Lisa Fiedler, Peizhu Qian, Vaibhav Unhelkar, Tina Seidel, Gjergji Kasneci, and Enkelejda Kasneci. Towards human-centered explainable ai: A survey of user studies for model explanations. TEEE Trunsactions on Paltern Analysis and Machine Intelligence, 2023.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.011000 2019.
Miriam Schirmer, Isaac Misael Olguín Nolasco, Edoardo Mosca, Shanshan Xu, and Jürgen Pfeffer. Uncovering trauma in genocide tribunals: An ulp approach using the genocide transcripi corpus. In Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, pp. 257-266, 266, 2023.
Cansu Sen, Thomas Hartvigsen, Biao Yin, Xiangnan Kong, and Elke Rundensteiner. Human attention maps for text classification: Do humans and neural networks focus on the same words? In Proceedings of the 58th annual meeting of the association for computational linguistics, pp. 4596-4608, 2020.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International conference on machine learning, pp. 3145 3153. PMLR, 2017.
Karen Simonyan, Andrea. Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint 1rXii:1312.600 6034, 2013.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg, Smoothgrad: removing noise by adding noise. arXiv preprint Xrviv: 706.00 708.03825. 2017.
Tianli Sun, Haonan Chen, Yuping Qiu, and Cairong Zhao. Efficient shapley values calculation for transformer explainability. In Asian Conference Pattern Recognition, pp. 54-67. Springer, 2023.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference machine learning, pp. 9319-3322 PMLR, 2017.
Mateusz Szezepański, Marek Pawlicki, Rafal Kozik, and Michal Choraš. New explainability method for bert-based model in fake news detection. Scientific reports, 11(1):23705, 2021.
Richard Tomsott, Dan Harborne, Supriyo Chakraborty, Prudhvi Gurram, and Alun Preece. Sanity checks for saliency metrics. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 6021-6029, 2020.
16

---

Published in Transactions on Machine Learning Research (12/2024)
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXie preprint AXi::2307.09288, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Shawn Xu, Subhashini Venugopalan, and Mukund Sundararajan. Attribution in scale and space. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pallern Recoguilion, pp- 9680 9689, 2020.
Sen Yang, Shujian Huang, Wei Zou, Jianbing Zhang, Xinyu Dai, and Jiajun Chen. Local interpretation of transformer based on linear decomposition. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10270-10287, 2023.
Chih-Kuan Yeh, Cheng- Yu Hsieh, Arun Suggala, David 1 Inouye, and Pradeep K Ravikumar. On the (in) fidelity and sensitivity of explanations. Advances in neural information processing systems, 32, 2019.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference artificial intelligence, volume 37, pp. 11121 11128, 2023.
Zihan Zhou, Mingxuan Sun, and Jianhua Chen. model-agnostic approach for explaining the predictions on clustered data. In 2019 IEEE international conference on data mining (ICDM), pp. 1528- 1533. IEEE, 2019.
17

---

Published in Transactions on Machine Learning Research (12/2024)
A
Motivation: Failure Cases For Model Recovery
We provide another motivational example that shows a failure case of current explanation methods on transformer architectures. In this example we test the recovery property for linear model. We create synthetic dataset where each word in sequence t has a linear contribution to the log odds score, formalized by
|e|
plu 1|6) ply 0ft)
log
40l) = b+ _ w ((4))
(6)
We create a dataset of 10 words (cf. Table 3 and train transformer models on samples from this dataset. We subsoquently create sequences that repeatedly contain single token т (in this case, r="perfect"), pass them through the transformers, and use Shapley values (approximated by Kernel-Shap) to explain their output. The result is visualized in Figure 7 and shows that a fully connected model (two-layer, 400 hidden units, RcLU) recovers the correct scores, whereas transformer models fail reflect the truc relationship. This shows that explanation methods that are explicitly implicitly based on additive models lose their ability recover the true data-generating process when transformer models explained.
expected
fully connected
4
seq. length 2c
N
2
4
6
8
SHIAP valuues
token
token
BERT
GPT-2
seq, length 0o
4
6
token
taken #
Figure 7: SHAP values do not recover linear functions F' for transformers. We compute SHAP values for token sequences that repeatedly contain a single token with a ground truth score of 1.5 (i.e., F(F)))1155, F(Ir, =3)== ...) such that the ground truth attributions should yield 1.5 independent of the sequence length. While this approximately holds for fully connected model, BERT and GPT-2 systematically overestimate the importance for short sequences and underestimate it for longer ones.
B Proofs
B.1
Formalization of the transformer
Many popular LLMs follow the transformer architecture introduced by Vaswani et al. 2017 with only minor modifications. We will introduce the most relevant building blocks of the architecture in this section. A schematic overview of the architecture visualized in Figure 2 Let us denote the input embeddings for
Layer E 1,... by (1-1) = [h] (11) ..... h U-1]TT E Rlt| where a single line h; contains the embedding for token The input embeddings of the first layer consist of the token emboddings, i.c., H))) E, where E [e,...., ] I E | is matrix of the individual token embeddings. At the core of the architecture lies the attention head. For each token, query, key, and value vector computed by applying
18

---

Published in Transactions on Machine Learning Research (12/2024)
Lffine-linear transform. In matrix notation this can be written as
(1) Q ((-1))
Q +1
6
(7)
K() H(111
W (1)
11 kk
(1)
(8)
v = H (( W (1) +1 b
where 1 11 E Ri4 denotes a vector of ones of length It, 6g b (l trainable parameters and da denotes the dimension of the attention head onlo each other and normalized by row-wise soflmax operation,
E Rdh W
K w E Raxd, are
Keys and queries projected
(Qu K dd
(0) rowsoftmax
(10)
This results in the attention matrix A E Rl where row i indicates how much the other tokens will contribute to its output embedding. To compute output embeddings, we obtain attention outputs 8;
s=
|18] = (1) (1)
(11)
Note that an attention output can be computed as s; = Σド Ui where v, denotes the value vector in the line corresponding to token j in V [u,...., m|]|]" and aij A The final 8; are projected back to the original dimension d by some projection operator P: Rd + Rd before they are added to the corresponding input embedding ((-1) due to the skip-connsstions The sum is then transformed by nonlinear function that we denote by [īn Rd - Rd. In summary, obtain the output for the layer, h with
h (1) = fh (-1) + P(8.)).
(12)
This procedure repeated iteratively for layers I,..., L such that we finally arrive at output embeddings ( To perform classification, a classification head cls E + R yl is put on top of token at classification index (how this token is chosen depends the architecture, common choices include r E (1,1t))) such
that we get the final logit output cls (1t The logit output is transformed to probability vector via another softmax operation. Note that in the two-class case, we obtain the log odds F(t) by taking the difference (A) between logits
p(y 1lt)
F(t) log
ply = 0 A(I) -1 lo-
(13)
B.2 Proof of Proposition 4.1
Proposition B.1 (Proposition 4.1 in the main paper). Let V be a vocabulary and C 2 2,C E N be marimum sequence length (contezt window). Let w, V - R, Vi E I,...,C' C' be any map that assigns token encountered al position i a numerical score including at least one loken T E V with non-zero weight wx(T) + 0 for some E 2....,C. C. Let b E R be an arbitrary offset. Then, there exists no parametrization of the encoder or decoder single-lager transformer F such that for every sequence t [t1, t,,..,,, Mee]] with length S [t| < C, the output of the transformer network is equivalent to
d tel) = b Σ (t.).
F(t1, t2.
(14)
Proof. We show the statement in the theorem by contradiction. Consider the token, 7 E V, for which Wy o 0 for some token index > which exists by the condition in the theorem. We now consider
We only formalize one attention head here, but consider the analogous caae multiple heads in our formal propfs.
19

---

Published in Transactions on Machine Learning Research (12/2024)
sequences of length & of the form tx
T for &o 1.... C. For example, have t [5]
repeat times
t [T, 7], etc. The output of the transformer is given by
F(t) - 9
h (0)
0r j=1
9 e(T)) Σ arjug
(15)
where is the token index on which the classification head is placed. Note that with r == |t] for the decoder architecture, the sum always goes up to It| (for the encoder architecture this is always true). As all tokens in the sequence have value of r, obtain h e(l,) e(7). The first input to the final part will thus be equal for all sequences tg. We will now show that the second part will also be equal.
We compute the value, key, and query vectors for т. v, k, Rd correspond to one line in the respective key, query and value matrices. As the inputs are identical and we omit positional embeddings this proof, lines are identical in the matrices. This results in
v wVee1 + bv
(16)
k e + +bK
(17)
q = WTe(r)+ + bQ
(18)
We omit the layer indices for simplicity. As pre-softmax attention scores (product of key and value vector), we obtain s = q k/vdg. Subsequently, the softmax computation will be performed over the entire sequence, resulting in
cxp(s) k exp((s)
= softmax s))
(19)
k times
(20)
à times
The second input Σ/ a, the feed-forward part is given by
1d Σ Urj =
k
Urjuj = Σ v
(21)
k
1=1
j=1
as arj and v are independent of the token index j. We observe that the total input to final part is independent of & in its entirety, as the first input e(r) independent of k and the second input is independent of ie well. As g is deterministic function, also the log odds output will be the same for all input sequences ts and be independent of k. By the condition we have non-zero weight wy(r) * for some j 2 2. In this case, there are two sequences f5-1 (length j-1) and tú (length j) consisting of only token т, where the outputs of the additive model (GAM) follow
faAM(t,)= bb Σ w((T)
(22)
5-1
=b+
Σ 1=1
wi(T) + wT(T)
(23)
= JCAM (t wz(T)
(24)
As we suppose (T(T) # 0, it must be that fGAM(t,) # fGAM(t)-1) which is a contradiction, with the output being equal for all sequence lengths.
20

---

Published in Transactions on Machine Learning Research (12/2024)
Multi-head attention. In the case of multiple heads, have
Ft)) A (cl((n
(25)
A cls [Tn(h,)) + Pa ( Pn=2(0t=2) +... Pr P (0#=H))
(26)
(h s
H=
(27)
As before, we can make the same argument, if we show that all inputs to are the same. This is straight- forward, as we can extend the argument made for one head for every head, because none of the head can differentiate between the sequence lengths. The first input will still correspond to h, (0) e(T), which results in the same contradiction.
7
B.3 Corollary: Transformers cannot represent linear models
Corollary B.2 (Transformers cannot represent linear models). Let the context window be C> and suppose the same model in Proposition 4.1 Let w - R be any weighting function that is independent of the token position with w(r) # O where for at least token E V. Then, the single layer transformer connot represent the linear model
F(11, (N]) b Σ w(t:)-
(28)
With w(r) * O, the condition from Proposition 4.1 i.e., having one wy with (x(r) A for 2 fulfilled as well such that the result of the proposition as well.
Proof. This can be seen by sctting w; = w for every i in Proposition 4.1
This statement has strong implication on the capabilities of transformers as it shows that they struggle to learn linear models.
B.4 Proof of Corollary 4.3
Corollary B.3 (Corollary 4.3 in the main paper). Under the same conditions in Proposition 4.1 slack of mulltiple transformer blocks in the model F neither has paramelrization sufficiend to represent the additive model.
Proof. We show the result by induction with the help of a lemma.
Lemma: Suppose a set S of sequences. If (1) for every sequence t E S the input motria H (h 1 will consist of impul embeddings that idenlical for each loken i, and (2) single inpul embeddings also have the same value for every sequence t E S, in the output H (1) the output embeddings will be identical for all tokens i and (2) they will have equal value for all the sequences E S considered before.
For the encoder-only architecture, the proof from Proposition 4.1 holds analogously for each token output embedding (in the previous proof, we only considered the output embedding at the classification token r). Without restating the full proof the main steps consist of
considering same-token sequences of variable length
showing the attention to be equally distributed across tokens, i.e. oij = 1/[t]
showing the value vectors v; to be equal because they only depend on the input embeddings which are equal
concluding that the output will be equal regardless of the number of inputs
21

---

Published in Transactions on Machine Learning Research (12/2024)
This shows that for each sequence t E S, the output at token i remains constant. To show that all tokens result. in the same output, we observe that the the only dependence of the input token to the output is through the query, which however is also equivalent if we have the same inputs.
For the decoder-only architecture, for token i, the attention weights are taken only up to index resulting in a weight of for each previous token and weight of (masked) for subsequent. ones. However, with the sum also being equal to and the value vectors being equivalent, there is no difference in the outcome. This proves the lemma.
Having shown this lemma, we consider a set S of two sequences S = (, where kz-1 contains j-1 repetitions of token 7 and t, contains j repetitions of token т. We chose j > 2, t such that w,(T) A 0, which is possible by the conditions of the theorem. We observe that for H the embeddinng are equal for each token and their value the same for both sequences. We then apply the lemma for layers 1.... L, resulting in the output embeddings of HL) being equal for each token, and most importantly identical for ビ and t,. As we perform the classification by F(t) A cls (n. (n. this output will also not change with the sequence length. This result can be used to construct the same contradiction as in the proof of Proposition 4.1
B.5 Proof of Proposition 5.1
Proposition B.4 (Transformers can easily fit SLALOM models). For any map 8, v and a transformer with an embedding size d, dn 2 3, there exists porameterization of the transformer reflect the SLALOM model in Equation
Proof. We can prove the theorem by constructing weight setup to reflect this map. We let the embedding e(r) be given by
e(r) = (()))),))),0,,...... 0].
(29)
We then set the key map matrix K to be
W₁- = 0
(30)
6y = [1, 0, 01.
(31)
such that we have
Whe(T) + bb [1,0,....,0
(32)
For the query map we can use
W 4 = I
(33)
b, = 0
(34)
such that
W.e(T) + b.
(35)
This results the non-normalized altention scores for query T E and key E
att,, (2) = (W,e(t:)+ b.) (W))et))++ + bb) = s(t;)
(36)
We see that regardless of the query token, the pre-softmax score will be s0).. For the value scores, perform similar transform with
W. diag ((0, 1,0. ...,.....
(37)
b, 0
(38)
22

---

Published in Transactions on Machine Learning Research (12/2024)
such that
v = W.e(t:) + b, (0,,,,
(39)
We then obtain
8, = Σ
ariV;
softmax:(s(t)))....
(40)
Nett
t
0,0,
Σ
(t)))(()))
(41)
Let
We saw that the final output can be represented by
F(t) A (cls (ffn (e(to) + P(s,))))
(42)
The projection operator is linear, which can set to easily forward in input by setting P= = Due to the skip connection of the feod-forward part, we can easily transfer the second part through the first ffn part. In the classification part, output the third component and zero by applying the final weight matrix
0
0 0
0
W class
(43)
0
and a bias vector of 0.
Multiple Heads. Multiple heads can represent the pattern by choosing I for one head and choosing P for the other heads.
Multiple Layers. We can extend the argument to multiple layers by showing that the input vectors can just be forwarded by the transformer. This is simple by sctting p= 0, the mill-map, which can be represented by linear operator. We then use the same classification hat before.
B.6 Proof of Proposition 5.2
Proposition B.5 (Proposition 5.2. in the main paper). Suppose query access to model G that takes sequences of tokens with lengths |t| I,...,C' and returns the log odds according to nor- non-constant SLALOM a vocabulary V with unknown parameter maps V + R v: - R. For C' 2 2, we can recover the true maps 8, with 2| - queries (forward passes) of F.
Proof. We first compute G([r)), Vr E V. We know that for single token sequences, all attention is on one token, ie., (a, = 1) and we thus have
G([T) o())
(44)
We have obtained the values scores for each token through [V] forward passes. To identify the token importance scores 8, we consider token sequences length 2.
We first note that if the SLALOM is non-constant and | > 1, for every token 7 E V, we can find another token 0 for which u(7) A v()). This can be seen by contradiction: If this would not be the case, i.e., we cannot find token with different value v(W), all tokens have the same value and the SLALOM would have to be constant. For |V| 1, SLALOM is always constant and does not fall under the conditions of the theorem.
We now select an arbitrary reference token f E V. We select another token 0 for which v(0) * v(0). By the previous argument such a token always exists the SLALOM is non-constant. We now compute relative importances w.r.t. 9 that we refer to as 17.. We let (0(T)) s(T) s(8) denote the difference of the importance between the importance scores of tokens 7, 0 V. We set n))))
We start with selecting token á and subsequently use each other token * to perform the following steps for each + 0:
23

---

Published in Transactions on Machine Learning Research (12/2024)
1. Identify reference token f. We now have to differentiate two cases: If (T)) v(0), select f = 1 the reference token. II u(T) # e(0), we select i = 0 as the reference token. By doing so, we will always have u(f) + v(T).
2. Compute G([T, f)). We now compute G([T, fl). From the model's definition, we know that exp(s(r))
G((T, exp(s(7)) + exp(s(f)) U(7)
(45)
exp(s(f))
exp(s(T)) + exp(s(f)) u(f)
(46)
-oddoo (()(s(e)) + exp(s(f)) G((r))
exp(s(T))
(47)
exp(s(f))
exp(s(r)) + cxp(s(()) G(FF)
(48)
exp(s(T))
exp(s(T)) exp(s<f)) G((Ti))
(49)
(
exp(s(T)) + exp(s(f)) G((+)
exp(s(T))
(50)
which we can rearrange
exp(s(T))
- exp(s(r)) + +xx((s(F) (G((7) G(()))
(51)
and finally to
cxp(s(T))
G([r, f)) G((f)
G([7)) - G(FF)) = g(T,f)
(52)
exp(s(r)) + exp(s(f))
and
= gTT,
(53)
+ exp()(f) exp(s(T)
g(T,f) = 1 exp(s(f))
(54)
exp(s(r))
+ log (g(
-1) = (T) - s(t) = d(r,
(55)
This allows us to express the importance of every token r E V relative to the base token A.
3. Compute importance relative to 0, i.e., mT()). In case we selected i 0, we set no(() d(T,7)
s(T) s(0). In case we selected & = 0, we set
(T) = d(T,, f) - d(0 0) = s(T) - -0) + (s =(0)) = =() (0)
(56)
The value of d(O, 0) is already known from the first iteration of the loop, where consider = 0 (and needs to be computed only once).
Having obtained a value of (T(T)) for each token 7 * 0, with |V| - 1 forward passes, we can then use the normalization constraint to solve for s0)) as in
(ne(T) + s())) =' 0 Σ TEV
(57)
such that we obtain
s(0) = Erev (1(7)
(58)
|V]
We can plug this back in obtain the values for all token importance scores s(7) s(0) (o().. We have thus computed the maps and in 2|V| forward passes, which completes the proof.
24

---

Published in Transactions on Machine Learning Research (12/2024)
B.7 Relating SLALOM to other attribution techniques.
Local Linear Attribution Scores. We can consider the lollowing weighted model:
F(2) Enee d (xp(((()))(()) Ei exxp(s(t)))
(59)
where A = 1 if a token present and A = it absent. We observe that setting X = 0 has the desired effect of making the output of the weighted model equivalent to that of the unweighted SLALOM on a sequence without this token.
Taking the derivative at = 1 results in
oF exp(s(t,))u(tt) (Er d, exp(s(t))))
(60)
(Ejee A exp(s(t;)))
exp(s(t.)) (Cienaine A exp(o(t)))e(t))
(61)
(Ei d exp(s(t))))
Plugging in A = 1, and using (t) = exp(a(t.))
we obtain
Zeyee A exp(a(t;))
OF
=
aa(()),)))))))) - (F(1) a.e(4)))
(62)
0 x=1
=a; ((v(4) (((()) a.,(4)))
(63)
a4
- (p(44)
(64)
Noting that a; exp(a(t.))
where R and F(1) are independent of i, we obtain
R
OF ax,
v(4,) exp((()))
(65)
A=1
which can be used rank tokens according the locally linear attributions. We refer to this expression linearized SLALOM scores ("lin").
Shapley Values. We can convert SLALOM scores to Shapley values Φ(i) using the explicit formula:
1 Σ n
("151) (FFS F(S))
(
(66)
SCINTEE
n |S| ) ((SU F(SU(i)) 0..2
Σ
(67)
1-
SCINIEE
n (n-1) 1)
(ailV: F(SU F(SU
Σ SC(N
(68)
n
|S|
a
a (v; - F(1))
(69)
1 04
However, computing this sum remains usually intractable, as the number of coalitions grows exponentially.
We can resort common sampling approaches CCatro et al. 2009 Malcki al. 2013) approximate the sum.
25

---

Published in Transactions on Machine Learning Research (12/2024)
C
Additional Discussion and Intuition
C.1
Generalization to multi-class problems
We can imagine the following generalizing SLALOM to multi class problems follows: We keep an importance map V + E that still maps each token to importance
previously. However, we introduce value score map V + IR for each class e y. Additionally to requiring
>S(T) =0.
s(T)
(70)
TEV
we now require
Σ cEV
(a([) 0,
(71)
For input sequence t, the SLALOM model then computes
Fo(t) log puy 1lt)
Σ
cn(())o(ts),
(72)
ply 0
TEE
The posterior probabilitics can be computed by performing a softmax operation over the F-scores, as in
exp(F.(t))
ply = clt) = Evey exp(Fe
(73)
We observe that this model has (1>) | - free parameters (for the two-class problem, this yields 2 - as before) and can fitted and deployed as the two-class SLALOM without major ramifications.
C.2 Practical Considerations
Our theoretical model contains slight deviations from real-world transformers to make it amendable to theoretical analysis. To represent token order, common architectures positional embeddings, tying the embedding vectors to the token position i. The behavior that we show in this work's s analysis docs however also govern transformers with positional embeddings for the following reason: While the positional embeddings could be used by the non-linear fin part to differentiate sequences different length in theory, our proofs show that represent the linear model, the softmax operation must be inverted for any input sequence. This highly nonlinear operation and the number of possible sequences grows exponentially at rate of M with the context length C. Learning-theorel considerations (e.g., Bartlett et al. 1998) show that the number of input-output pairs the two-layer networks deployed can maximally represent is bounded by O (dMnidden log (og(ddiid)))) )). which is small (d=786, -3dd for BERT) in contrast to the number of sequences (C = 1024, [V] x x 104). We conclude that the inversion is therefore impossible for realistic setups and positional embeddings can be neglected, which is confirmed by our empirical findings.
Common models such as BERT also use a special token referred as CLS-token where the classification head is placed on. In this work, we consider the CLS token just as a standard token in our analysis. In our empirical sections, always append the CLS token mandated by the architecture to make the sequences valid model inputs.
Algorithm: Local SLALOM approximation
We propose two algorithms to compute local explanations for a sequence [t1....,ttp] tpei] with SLALOM scores. particular, we use the Mcan-Squarod-EFrr (MSE) fit SLALOM on modified soquences consisting of the individual tokens in the original sequence t. To speed up the fitting we can sample a large collection of samples offline before the optimization.
26

---

Published in Transactions on Machine Learning Research (12/2024)
D.1 Efficiently fitting SLALOM with SGD
For the elficient implementation SLALOM-eff given in Algorithm | we sample minibatches from this collection in each step and perform SGD steps on them. We perform this optimization using b = 5000 samples in this work. We use sequences of n = 2 random tokens from the sample for SLALOM-eff, making the forward passes through the model highly efficient.
D.2 Fitting SLALOM through iterative optimization
For the high-fidelity implementation SLALOM-fidel (Algorithm 2 we first use a different set of sequences and model scores fit. the surrogate: We delete up to tokens from the original sequence randomly create the estimation dataset (similar to LIME). The fitting algorithm optimized for maximum fidelity uses iterative optimization scheme to fit SLALOM models. It works by iteratively fitting and to the dataset obtained. Denote by E TR the model scores obtained for the b input sequences 4, 1...b. As the SLALOM model in Equation (4) is a linear combination of the values score weighted by the normalized exp(s(t.)) importance score, can set up a matrix A, where element dijj E Eijer, exp(s((,)) provides the normalized
importance for given s. We solve
(An
(74)
for v, which is linear ordinary least squared problem that be solved through the normal equation. This
results in the optimal v for the given In second step, we keep v fixed and find better scores. We can reformulate the equations for the samples as
Σ exp(s(()))(()))
Eex exp((((s))) fi
(75)
ty
EEE
Σ
exp(s(t))(((tt)) - fi)
(76)
exp
This problem be written with vector ā E R and matrix E E R and results in optimization problem
min( Ex) (E3).
(77)
s.t. á $20
(78)
| 2 |
(79)
The conditions ensure that we can obtain the original s-scores as log (element-wise) and that the trivial solution & = 0 is not assumed. We solve this problem using solver implemented in scipy. optiyyzeptmmzz..... ceast_squares
E
Experimental Details
In this section, we provide dotails on the experimental sctups. We provide the full source-code our GitHub repository.
E.1
Fitting transformers on synthetic dataset
E.1.1 Dataset construction: Linear Dataset
We create a synthetic dataset to ensure linear relationship between features and log odds. Before sampling the dataset, we fix vocabulary tokens, ground truth scores for each token, and their occurrence probability. This mcans that each of the possible tokens already comes with ground-truth score w that has boen manually assigned. The tokens, their respective scores w, and occurrence probabilities are listed in Table 3 Samples of the dataset are sampled in four steps that exectued repeatedly for each sample:
27

---

Published in Transactions on Machine Learning Research (12/2024)
Algorithm Local efficient SLALOM approximation (SLALOM-eff)
Require: Sequence i, trained model F' (outputs log odds), random sample length n, learning rate A, batch size r, sample pool size b, number of steps c Initialize v(t,) = 0, s(t;) = 0 A unique t, E + samples of random sequences of length n obtainod through uniform sampling of unique tokens in
Precompute F 1,...,b b # perform model forward-pass for each sample in pool
steps f- 0
while steps < do B' t minibatch of samples uniformly sampled from the sample pool B ,.ALOM,, loss +- + E (SAA (B(( # compute MSF blu. F and SLALOOM using precom-
puted models outputs F(B")
# AV ,ooss # pook-propegate loss to update SLALOM porameters
$ - XV losss
steps steps +1
end while return mean(s) # normalize to zero-mean
Algorithm Local high-fidelity SLALOM approximation (SLALOM-fidel)
Require: Sequence t, trained model (outputs log odds), max. number of deletions n, learning rate X, batch size r, sample pool size b, number of steps c Initialize 0, s
unique t; E
B 4- samples of random sequences of length obtained through deleting up to tokens randomly from Precompute F(B b,...,b # perform model forward-pass for each sample in pool
steps +- 0
while steps < do v = arg min.' 0' 2 (F(B(()) - s # Fix and optimize v, OLS problem
arg ming D - (F(B()) o. (B(i))² # Fir and oplimize 8, Quadratic problem
steps < steps +1
end while return v, 8 - mean(s) # normalize 8 to zero-meon
1. A sequence length |t| Bin(p n=30) is binomially distributed with an expected value of 15 tokens and maximum of 30 tokens
We sample |t] lokens independently from the vocabulary according to their occurrence probability (Table 3h
Third, having obtained the input sequence, we can evaluate the linear model by summing up the scores of the individual tokens in sequence:
F(t) F(11, (0))
Σ w(1)).
(80)
Having obtained the log odds ratio for this sample F(t), we sample the labels according to this ratio. We have p(y = 1)/p(y 0) = exp(F(t)), which can be rearranged to ply = 1) I+exp(F(E)) sample binary label for each sample according to this probability.
exp(F(e))
We
The tokens appear independently with the probability Poccurrence given in the table.
E.1.2 Dataset Construction: SLALOM dataset
We resort to second synthetic dataset to study the recovery property for the SLALOM. This required a SLALLOM relation between the data features and the labels. To find realistic distribution of scores, we
28

---

Published in Transactions on Machine Learning Research (12/2024)
word "the"
"we"
"movie"
"watch"
"good"
"best"
"perfect""
"ok"
"bad"
"worst."
linear weight w
0.0
0.0
0.0
1.0
1.5
-0.6
-1.0
-1.5
Poceurrence
1/6
1/6
1/6
1/6
1/15
1/20
1/20
1/15
1/20
1/20
Table 3: Tokens in the linear dataset with their corresponding weight
compute BoW importance scores for input tokens of the BERT model on the IMDB dataset by counting the class-wise occurrence probabilities. We select 200 tokens randomly from this dataset. We use these scores as value scores v but multiply them by a factors of as many words have very small BoW importances. realistic datascts, we observed that value scores v are correlated with the importance scores s. Therefore, we sample
((T) (7))5((()))) + N (0.1),
(81)
which results in the value/importen /importace distribution given in Figure We assign each word an equal occurrence probability and sample sequences of words at uniformly distributed lengths in random [1, 30). After sequence is sampled, labels subsequently sampled according to the log odds ratio of the SLALOM.
E.2 Post-hoc fitting of surrogate models
We train the models on this dataset for epochs, where one epoch contains 5000 samples at batch size of 20 using default parameters otherwise.
For the results in Figure 3 we query the models with sequences that contain growing numbers of the work perfect, ie. ["perfect", "perfect", ...]. We prepend CLS token for the BERT models.
For the results in Figure b),, we then sample 10000 new samples from the linear synthetic dataset (having the same distribution as the training samples) and forward them through the trained transformers. The model log odds score together with the feature vectors are used to train the different surrogate models, linear model, GAM, and SLALOM. For the linear model, we fit an OLS on the log odds returned by the model. We use the word counts for each of the 10 tokens each token different weight according to its position in the sequence. To this end, we vector of length 30 10. Each feature corresponds to token and position and is set to one if the token i is present at this position, and set to 0 otherwisc. We then fit lincar model using regularized (LASSO) least squares with a small regularization constant of X = 0.01 because the system is numerically unstable in its unregularized form.
feature vector. The GAM provides the possibility
assign different feature
E.3
Recovering SLALOMs from observations
To obtain the results in Figure [c,d), we train the transformer models 20 - 20000 samples from the second syuthelic dataset (SLALOM dalaset). When using smaller vocabulary size, only sample the sequences out of the first [V] possible tokens (but keeping the ground truth scores identical).
E.4 Training Details for real-world data experiments
Training details. In these experiments, we use the IMDB MMas et al., 2011 ) and Yelp (Asghar 2016 datasets to train transformer models on. Specifically, the results in Table la are obtained by training 2-layer versions of BERT, DistilBERT and GPT-2 with on 5000 samples from the IMDB dataset for 5 epochs, respectively. We did not observe significant variation in terms of number of layers, so we stick to the simpler models for the final experiments. For the experiments in Table lb we train and use 6-layer versions of the above models for 5 epochs on 5000 samples of the Yelp dataset. We report the accuracies of these models in Table ! and additional hyperparameters in Table
SLALOM vs. Naive Bayes Ground Truth. To arrive the Spearman rank-correlations between SLALOM importance scores 8, value scores and their combination (exp(s) -)) with ground truth, fit.
29

---

Published in Transactions on Machine Learning Research (12/2024)
importance score
5
0
2
value score
Figure 8: Score distribution for the tokens for the analytical SLALOM used in the recovery experiment.
dataset
DistilBERT
BERT
GPT-2
IMDB
0.88
0.90
0.74
Yelp
0.86
0.88
0.88
Table Accuracies of models used in this paper. For IMDB ([trainset) 5000), we use 2-layer versions of the models. For Yelp, ([trainset] 5000), we use 6-layer versions of the models. For both datasets, the [testset] 100. The models are trained for 5 epochs after which we found the accuracy of the model on the test set to have converged.
SLALOM on each of the trained models and use a Naïve-Bayer model for ground truth scores. The model is given as follows:
p④y 1|t) p(y 1)
p( y 1)
log
log
+
log
(82)
p④y = 0lt) p = 0)
1. p(t. 0)
We obtain p(t.y==1)
p(t/o=0) by counting class-wisc word frequencics, such that we obtain lincar score w for cach token given by w(r) (#oce. of in class 1)+a We use Laplace smoothing with 40. The final correlations are computed over
(#foee in D)+aa
of 50 random samples, where we observe good agreement between the Naäve Bayes scores, and the value and linearized SLALOM scores, respectively. Note that the importance scores are considered unsigned, such that compute their correlation with the absolute value of the Naive Bayes scores.
SLALOM vs. Human Attention. The Yelp Human Attention (HAT)) Sen et al. 2020 dataset consists of samples from the original Yelp review dataset, where for each review real humman annotators have been asked to select the words they deem important for the underlying class (i.e. positive or negative). This results in binary attention vectors, where each word either or not important according to the annotators. Since each sample is processed by multiple annotators, we use the consensus attention map as defined in [en al. (2200), requiring agreement between annotators for token to be considered important to aggregate them into attention vector per sample. Since HAT, unlike SLALOM, operates word's human attention to each of its tokens (according to the employed model's specific tokenizer).
word level, map each
To compare SLALOM scores with human altention in Table [b] we choose the AU-ROC metrie, where the binary human attention serves as the correct class, and the SLALOM scores as the prediction. We observe how especially the importance scores of SLALOM are reasonably powerful in predicting human attention. Note that the human attention scores are unsigned, such that we also use absolute values for the SLALOM value scores and the linearized version of the SLALOM scores for the HAT prediction.
30

---

Published in Transactions on Machine Learning Research (12/2024)
parameter
value
learning rate
50-5
specification
value
batch size
5
epochs
5
CPU core:
AMD EPYC 7763
dataset size used number of heads
5000
Num. CPU cores
64-Core (128 threads)
12
GPU type used
IxNvidia A100
number of layers
2 (IMDB), 6 (Yelp)
GPU-RAM
80GB
num. parameter
31M 124M
Compute-Hon Hours
150
(a) Hyperparameters
(b) Hardware used (internal cluster)
Table 5: Overview over relevant hyperparameters and hardware
architecture
L (num.layers)
linear model
GAM
SLALOM
GPT-2
20.31 + 2.02
48.78 + 2.70
16.92 = 1.33
GPT-2
24.81 + 3.11
54.33 3.26
22.17 = 1.98
GPT-2 6
32.66 1 7.60
57.08
7.19
21.59 = 4.14
GPT-2
12
25.74 + 4.18
54.36 + 3.94
20.25 2.37
DistilBERT
28.28 + 4.30
44.43 + 2.22
10.83 = 2.13
DistilBERT
32.58 1 7.75
53.87 1 7.20
16.82 = 4.38
DistilBERT
31.49 + 4.06
49.35
3.13
17.26 = 3.64
DistiBBERT 12
50.82 = 9.21
9.19 71.64
27.50 = 4.18
BERT 1
26.33 + 1.90
43.30 + 0.88
7.34 + 0.70
BERT
28.43 1 3.75
48.28
3.23
9.92 1 1.19
BERT 6
50.82 + 6.34
68.23
+ 4.59
23.99 = 3.38
BERT
44.58 13.15
51.38
14.71
18.77 = 6.78
Table 6: MSE (x100) when fitting SLALOM to the outputs of transformer models trained on the linear datasct. SLALOM manages to describe the outputs of the transformer significantly better than other surrogate models even if the underlying relation in the dala was linear.
F Additional Experimental Results
F.1 Fitting SLALOM as a Surrogate to Transformer outputs
We provide an additional emprical counterexample for why GAMs cannot describe the transformer output in Figure [ The example provides additional intuition for why the GAM insufficient to describe transformers acting like a weighted sum of token importances.
We report Mean-Squared Errors when fitting SLALOM to transformer models trained the linear dataset. in Table 6 These results underline that SLALOM outperforms linear and additive models when fitting them to the transformer outputs. Note that even if the original relation in the data was linear, the transformer does not represent this relation such that the SLALOM describes its output better. We present additional qualitative results for other models Figure 10 that support the same conclusion.
F.2 Fitting SLALOM on Transformers trained on data following the SLALOM distribution
We report Mean-Squared Errors in the logil-space and the parameter-space between original SLALOM scores and recovered scores. The logit output are evaluated on a test set of 200 samples that are sampled from the original SLALOM. We provide these quantitative results in Table 7 in Table R for the parameter space. In logits the differences are negligibly small, and seem decrease further with more layers. This finding highlight. that a) transformers with more layers still easily fit SLALOMS and such model be recovered in
31

---

Published in Transactions on Machine Learning Research (12/2024)
Input Sequence
BERT score
GAM weight assignments
SLALOM weight assignments
perfect
2.5
Assign wy ("perfect") 2.5
Assign e("perfect") = 2.5
perfect
perfect
2.5
Assign W2(" wx("perfect") =
Expected SLAT.OM output: 2.5
worst
-2.6
Assign wi("""wwwww" ("worst")
-2.6
Assign e("worst") -2.6
worst
worst
-2.5
Assign wy("worst") 0.1
Expected SLALOM outpul: -2.6
worst
perfect
0.0
Expected GAM output
Assign s (("perfect") = s a("worst") =
Wh ("perfect") ") + wy( ("w"rsst"") = 2.6 Expected SLALOM output: -0.05
Figure 9: A simple empirical countercxample for why GANs cannot describe transformer output. We report rounded scores by a real 4-layer BERT model (similar behavior was observed for other layers/architecllres) and iteratively fit the GAM F(t) = w; to match observed outputs on the two tokens "perfect" and "worst". We quickly arrive at a contradiction for the GAM. On the contrary, we can assign SLALOM scores that model this behavior with minor error. Because transformers behave like weighted sum of importances, GAMs are insufficient to model their behavior. In conjunction with Figure Na.b) this underlines that GAMs and linear models are insufficient as surrogates.
L (num layelsyers)
DistilBERT
BERT
GPT-2
0.002 + 0.001
0.002 + 0.000
0.011 + 0.009
0.003 = 0.002
0.003 = 0.002
0.017 + 0.011
0.001 + 0.001
0.011 1 0.007
<0.001 + 0.000
12
<0.001 + 0.000
<0.001 + 0.000
<0.001 + 0.000
Table 7: MSE (x100), logit space, averaged over 5 runs
parameters space. The results on the MSE in parameter space show no clear trend, but are relatively small as well (with the largest value being MSE=0.015 (note that results in the table are multiplied by a factor of 100 for readability). Together with our quantitative results in Figure (c,d), this highlights that SLALOM has effective recovery properties.
F.3 Additional Results on Real-World Data
We obtain SLALOM explanations for real-world data using the procedure outlined in Algorithm 1 (SLALOM- eff) with sequences of length and with Algorithm 2 (SLALOM-fidel) removing up to tokens that compare with Naive-Bayes scores and Human Attention.
F.3.1 Additional Qualitative Results
Figure 12 shows the full results from the sample used in Figure 5 where we only visualized a choice of words for readability purposes. After running SLALOM-eff on our trained IMDB models, we use to explain a movie review laken from the dataset, visualizing value scores against importance scores 8.
L (num.layer)
DistiLBERT
BERT
GPTT2
0.092 + 0.045
0.540 + 0.301
0.940
0.392
2
0.094 + 0.049
0.368 + 0.085
1.652
0.903
6
0.124 = 0.030
0.830 = 0.182
0.569
0.177
12
0.287 + 0.088
0.394 1 0.255
0.385 + 0.126
Table 8: MSE (x 1000), parameter space, averaged over 5 runs
32

---

Published in Transactions on Machine Learning Research (12/2024)
GAM
20
GAN
lindd
real DisiBERT
log oatd
SLALOM
log-volk
rod DiciBEXT
15
SLALOM
Log
15
10
15
10
15
(WOCC, of perflctt))
"perfoct"
"periect's
"perfect"
(a) DistilBERT (T=2)
(b) BERT (T.=12)
(c) DistilBERT (T==12)
(d) GPT-2 (T==12)
Figure 10: SLALOM describes outputs of transformer models well. Fitting SLALOM the outputs of the models shown in Figure B using the synthetic dataset. We show results for sequence containing repetitions of the token "perfect". Note however that the models were trained on larger dataset. of random sequences samples described in Appendix E.1.I but these sequences were chosen for visualization purposes. Results on additional models. Despite having C/2-15x more parameters than the SLALOM model, the GAM model does not describe the out as accurately. We provide quantitative results in Table 6
SLALOM-fidel
SLALOM-eff
LM
values
importances
SLALOM
values
importances
lin. SLALOM
DistilBERT
0.602 0.10
0.020 0.08
0.602 + 0.10
0.692 0.05
0.373 0.09
0.693 +
BERT
0,175
0.031
0.09
0.474
0.619
0.349 0.08
0.626 =
GPT-2
0.467 0.17
0.017 0.08
0.468 + 0.17
0.618
0.08
0.292
0.10
0.619 + 0.08
LM
LIME
SHAP
IG
Grad
LRP
Distilbert
0.691
0.05
0.619 = 0.06
-0.285
0.12
-0.215 = 0.12
0.706 0.05
BERT
0.616
0.08
0.554 + 0.09
-0.125
+ 0.14
-0.123 + 0.14
0.639 + 0.08
GPT2
0.213 0.13
0,500 = 0.09
0,033 + 0.13
0.031 = 0.13 0.615 :: 0.08
Table 9: Correlation with linear Naive Bayes Scores. The scores obtained with SLALOM-eff (value, lin.) are better than those from LIME, SHAP and comparable to LRP scores.
F.3.2 Correlation with Naive-Bayer Scores
We compare the scores obtained with SLALOM with the scores obtained with other methods in Table J obtaining scores that are reliable with SLALOM-eff (value scores and linear scores) in particular. While SHAP achieves higher correlation on BERT, SLALOM achieves higher correlation than LIME and SHAP all models and higher correlations than LRP for GPT-2 while obtaining slightly inferior values for the BERT-based architectures.
F.3.3 Human Attention
In Figure we show qualitative results for sample from the Yelp-HAT dataset. After fitting SLALOM on top of the resulting model, we can extract the importance scores given to each token in the sample. We can see that the SLALOM scores manage to identify many of the tokens which real human annotators also deemed important for this review to be classified as positive. We also show qualitative results for the other methods. However, we suggest cantion when interpreting explanations visually without ground truth. We argue that (1) theoretical properties of explanations (2) comparing to known ground truth as well (3) consideration of metrics from different domains, e.g., faithfulness, human perspective, are required to allow for comprehensive evaluation. This is the approach taken in work.
We show quantitative comparison of the scores obtained with SLALOM with the scores obtained with other methods on the comparison with Human-Attention in Table 10
F.4 Insertion and Removal Benchmarks
It important to verify that SLALOM scores are competitive to other methods in classical explanation benchmarks as well. We therefore the classical removal and insertion benchmarks with ST.AL.MM
33

---

Published in Transactions on Machine Learning Research (12/2024)
the
most delicious Steak
ve
ever
had
also the most the most delicious
Steak
ve
ever
had
also
the
most
expensive but it
was
tota lly
worth
it
our
waiter
was expensive
but
was
totally
worth
our
wa
iter
was
incredible was
his assistant
and
we
loved
the
vibe
incred
as
was
his
assistant
and
lved
the vibe
cocktail not
the restaurant
stuff
really
fun
great
restaurant
stuff
really
great
ever
back
vegas
ooe
to
cocktai
m
ever
back
in
vegas
love bo
return
return
(a)
Human annotation
(b) SLALOM-eff
the
most delicious ous
steak
vevervv
had
|lso
the most| the most del (doliclous
steak
||]|
ve
lever
had
a
the
most
expensive but it
was
totaly
worth
it
walter was
expensive
but
was
tota
worth
it
our wa
iter
was
incredible his
assistant
the vibe incredible
his
assistant
loved
vibe
restaurant
not
100
stuff
reall
fun
great
the restaurant
not
too
stuff
real
fun
great
cocktail m
ever
back in
vegas
ove
to
cocktail
ever back
in
vegas
d
love
b0
return
return
(c) SHAP
(d) LIME
the most
del ous
Steak
had
also the
most
expensive
but
was
lyy
worth
waiter
incredible
35 was
assistant
and
we
oved the
vibe
of
the
restaurant not 100 stuff
really
fun great
cocktail m
ever
back
In
vegas
lve
1o
return
(e) AttnLRP
Figure 11: Qualitative comparisons of attribution maps. We provide attribution maps for the different techniques in this figure. Many words deemed important by human annotators are likewise highlighted by SLALOM and other techniques.
2.0
2..
wpwaaa 5318
3.8
3.5
3.9
-25
-2
valut score
(a) BERT
(b) GPT-2
Figure 12: Full scatter plots of SLALOM scores for the sample shown in the main paper (please zoom in for details). We observe that words like "bad" or "fun" get assigned high importance scores and value scores of high magnitude (albeit with different sigus) by SLALOM.
34

---

Published in Transactions on Machine Learning Research (12/2024)
LM
valtiss e
importances
lin.
LIME
SHAP
LRP
Bert
0.796 + 0.01
0,807 # 0.01
0.801
0.01
0.803 = 0.01
0.800 0.01
0.813 0.01
Distil-BERT
0,688 + 0.01
0.681 + 0.01
0.686
0.01
0.702 + 0.01
0.668
0.01
0.703 + 0.01
G
0.674 a 0.01
0.685 a 0.01
0.683
0.01
0.632 4 0.01
0.671 0.01
0.699 1: 0.01
Table 10: Comparison of techniques to predict Human Attention. SLALOM-eff (importances) perform better than SHAP, comparably to LIME and slightly below TRP.
SLALOM-ridel
SLALOM-efE
LM
1-8001
lin.
0-scores
lin.
LIME
SHAP
IG
Grad
LRP
BERT 0.893 0.012
0.901 + 0.012 0.881 + 0.010 0.885
0.010
0.875
0.012
0,881
0.084 4 0.010 0.069 1 0.008
0.852 0.019
DistiBEERT 0,841
0.014 0,854 = 0.013
0.888 + 0.008
0.886
0.008
0.838
0.012
0,864
0.009
0.143 + 0.012
0.131 + 0.012
0.865 0.011
GPTT-2 0,837 # 0.013 0.814 # 0.013
0.752 + 0.013
0.784 + 0.012
0.479
0.021
0,859 # 0.012
0,289 + 0.021
0.269 = 0.020
0.833
0.013
average 0.837 0.013 0.866 + 0.013 0.851 + 0.010 0.852 0.010 0.731 0.016 0.868 0.011 0.172 + 0.014 0.156 + 0.013
0.850 0.014
(a) IMDB: Area-Over Perturbation Curve (delction, higher is better)
SLALOM-ridel
SLA
LM
1-800ms
lin.
0-scores
lin.
LIME
SHAP
IG
Grad
LRP
BERT 0.015 + 0.005 0.011 + 0.005
0.011 + 0.005
0.011 + 0.005
0.017
0.009
0.012 = 0.005
0.224 + 0.024
0.214 + 0.022 0.010 = 0.005
DistiBERT
0.018
0.006
0.019 1 0.006
0.082 1 0.009
0.032
0.009
0.014
0.005
0.020
0.005
0.250
1 0.027
0.249 1 0.026
0.017
0.000
GPT-2
0.033
0.032
0.007
0.015 + 0.007
0.015
0.007
0.129
0.021
0.001
0.251 # 0,021
0.214 = 0.021
0.039
0.007
average 0.022 + 0.006 0.021 + 0.006 0.029 + 0.007 0.029 + 0.007 0.063
0.011
0.018 + 0.005 0.242 + 0.025 0.236 + 0.024
0.022 = 0.007
(b) Yelp: Area-Over Perturbation Curve (insertion, lower is better)
SLALOM ridel
SLALOM
LM
1-800me
lin.
0 0-scores
lin.
LIME
SHAP
IG
Grad
LRP
GPT-2 0.747 # 0.024
0.753 + 0.024 0.726 + 0.021 0.727 + 0.021
0.444
0.028
0.849 + 0.015
0.292 + 0.026
0.290 + 0.026
0.740 = 0.025
BERT
0.637 4: 0.038
0.667 4: 0.038
0.865 1: 0.012
0.863 4: 0.013
0.797
0.022
0.859 4: 0.013
0.249
4 0.028
0.281 4: 0.029
0.855 :: 0.017
DistiBERT
0.645 4 0.033
0.641 = 0.033
0.813 + 0.017
0.813 :: 0.018
0.746
0.025
0.854 # 0.013
0.201 + 0.026
0.213 : 0.028
0.768 = 0.021
average 0.683 1 0.032
0.687 1 0.032 0.801 1. 0.017 0.801 1. 0.017 0.663
0.025
0.854 1 0.014 0.247 1. 0.027 0.271 1. 0.028
0.788
0.022
(c) Yelp: Area-Over Perturbation Curve (deletion, higher is better)
Table 11: Additional results for removal/insertion tests: We show results on the IMDB dataset for removal as well as insertion and removal on the Yelp datset.
compared to baselines such as LIME, SHAP, Grad (Simonyan et al. 2013 and Integrated Gradients (IG, Sundararajan et al. 2017). For the insertion benchmarks, the tokens with the highest attributions are inserted to quickly obtain a high prediction score to the larget class. For the deletion benchmark, the tokens with the highest attributions are deleted from the sample to obtain a low score for the target class. We subsequently delete finsert more tokens and compute the "Area Over the Perturbation Curve" (AOPC) as in Young ct al. 20200, which should be high for deletion and low for insertion. In addition to the insertion results in Table Tc the removal results are shown in Table 11a] We show results for the Yelp dataset in Table [1b and
Table Llc We claim that linear SLALOM scores perform on par with LIME and SHAP in this benchmark,
but do not always outperform them in this metric. For surrogate techniques (LIME, SHAP, SLALOM) we
use 5000 samples each.
F.5 Error Analysis for non-transformer models
We also investigate the behavior of SLALOM for models that do not precisely follow the architecture described in the Analysis section of this paper. In the present work, we consider an attribution method that is specifically catered towards the transformer architecture, which is the most prevalent in sequence classification. We advise caution when using our model when the type of underlying LM is unknown. In this case, model-agnostic interpretability methods may be preferred.
However, we investigate this issue further: We applied our SLALOM-eff approach to a simple, non-transformer sequence classification model the IMDB dataset, which is three-layer ferd-forward network based
35

---

Published in Transactions on Machine Learning Research (12/2024)
SHAP
LIME
lin. SLALOM-eff
Insertion (lower)
0.0120
0.005
0.0053 + 0.003
0.0039 + 0.001
Deletion (higher)
0.8022
0.026
0.9481 = 0.005
0.9601 + 0.004
Table 12: AOPC explantion fidelity metrics for the Fully Connected TF-IDF model. The scores highlight that SLALOM can also provide faithful explanations for non-transformer models due to its general expressivity.
(a) DistiBBERT
Approach / # samples
1000
2000
5000
10000
SHAP
2.35 + 0.01
4.62 + 0.02
11.56 = 0.03
23.08 + 0.08
LIME
0.80 1 0.04
1.58 1 0.07
3.93 1 0.19
8.04 + 0.39
SLALOM-fidel
0.74 + 0.03
1.42 + 0.06
3.77 + 0.24
7.95 = 0.41
SLALOM-eff
0.42 0.01
0.80 0.01
2.03 0.01
4.13 0.02
LRP
0.02 0.00
0.02 0.00
0.02 0.00
IG
0.02 + 0.00
0.02 1 0.00
0.02 0.00
0.02 + 0.00
Grad
0.01 + 0.00
0.01 + 0.00
0.01 + 0.00
0.01 + 0.00
(b) BLOOM-7B
Runtime (s)
SHAP
LIME
IG
Grad
SLALOMMeff
1000 Samples 25.04
2.37
15.73 2.37 0.29 + 0.04 0.06 0.01
0.62 0.02
Runtime results for computing different explanations. SLALOM is substantially more efficient that other surrogate model explanations (c.g., LIME, SHAP). Gradient-based explanations be computed even quicker, but are very noisy and require backward passes. Runtimes are given in seconds (s).
Table 13:
TF-IDF representation of the imputs. We compute the insertion and deletion Arra-over-perturaatiioccmum metrics that are given in Table 12
These results show that due to its general expressivity, the SLALOM model also succeeds provide explanations for non-transformer models that outperform LIME and SHAP in the removal and insertion tests. We also invite the reader to confer Table Tb. where we show that SLALOM can predict human attention for large models, including the non-transformer Mamba model
Gu & Dao 2023).
F.6 Runtime analysis
We ran SLALOM well as other feature attribution methods using surrogate models and compared their runtime to explain single classification of 6-layer BERT model. We note that the runtime is mainly determined by the number of forward passes to obtain the samples to fit the surrogates. While this number is independent of the dataset size, longer sequences require more samples for the same approximation quality. The results are shown in Table 13
While IG and Gradient explanations are the quickest, they also require backward passes which have large memory requirements. As expected, the computational complexity for surrogate model explanation (LIME, SHAP, SLALOM) is dominated by the number of samples and forward passes done. Our implementation of SLALOM is around 2x faster than LIME and almost 5x faster than SHAP (all approaches used GPU-based, batching-cnabled implementation). which we attribute to the fact that SLALOM can be fitted using substantially shorter sequences than are used by LIME and SHAP.
We are interested to find out how many samples are required to obtain an explanation of comparable quality to SHAP. We successively increase the number of samples used in the deletion benchmark (where the prediction should drop quickly when removing the most important tokens). We report the Area over the Perturbation Curve (AOPC) as before (this corresponds to their
our surrogates and report the performance
36

---

Published in Transactions on Machine Learning Research (12/2024)
Number of samples
Deletion AOPC
SHAP (nsamplos="anto")
0.9135 + 0.0105
SLALOM, 500 samples
0.9243 + 0.0105
SLALOM, 1000 samples
0.9236 + 0.005
SLALOM 2000 samples
0.9348 + 0.005
SLALOM, 5000 samples
0.9387 + 0.005
SLALOM, 10000 samples
0.9387 + 0.005
Table 14: Ablation study on the number of samples required to obtain good explanations. The results highlight that a number as low as 500 samples can be sufficient to fit. the surrogale model at a quality comparable to SHAP.
Comprehensiveness metric of ERASER YDe al., 2020 higher scores are better). We compare the performance to shap KernelExplainer shap_valuee values Insamples=ause method of the shap package in Table 14 Our results indicate that sampling sizes as low as 500 per explained instance (which is as low predicted by our theory, with average sequence length of 200) already yields competitive results.
F.7 Applying SLALOM Large Language Models
Our work mainly concerned with sequence classification. In this application, we observe mid-sized models like BERT be prevalent. On the huggingface hub, among the 10 most downloaded models on huggingface, are BERT-based and the remaining one is another transformer with around 33M parameters (as of September 2023). In common benchmarks like DBPedia classification[] the top-three models are transformers with two of them also being variants of BERT. We chose our experimental setup to reflect this. Nevertheless, we are interested to see if SLALOM can provide useful insights for larger models as well and therefore experiment with larger models. To this end, we use a model from the BLOOM family (Le Scao et al., 2023) with 7.1B parameters as well as the recent Mamba model (2.8B) (Gu & Dao 2023) on the Yelp-HAT dataset and compute SLALOM explanations. Note that the Mamba model does not follow the transformer framework considered in this work. We otherwise follow the setup described in Figure and assess whether our explanations predict human attention. The results the bottom of Table Ib highlight that this is indeed the case, even for larger models. The ROC scores are in a range comparable to the ones obtained for the smaller models. For the non-transe non-transformer Mamba model we observe a drop in the value of the importance scores. This may suggest that value scores and lincarized SLALOM scores are more reliable for large, non-transformer models.
Applying SLALOM to blackbox models. Finally, we would like emphasize that SLALOM, as surrogate model can be applied to black-box models well. To impressively showcase this, we apply SLALOM to OpenAl's GPT-4 models via the API (Appendix F.7). We use the larger GPT-4-turbo and smaller GPT-4o-mini for comparison. We prompt the model with the following template to classify the review and only output either 0 or 1 as response.
You are assessing movie reviews in an online forum. Your goal is to assess the reviews' overall sentiment as 'overall negative' (label '0') or 'overall positive' (label '1'). Your will see review now and you will output label. Make sure to only answer with either '0' or T. <the review>
SYSTEM:
USER:
We then use the token probabilities returned by the API to compute the score. We create 500 samples training dataset for the surrogate model and fit SLALOM the model using SLALOM-eff. We obtain the importance plots shown in Figure 13. This highlight that SLALOM scales up to large models. However, we would like to stress that there can be no guarantees as we have no knowledge about the specific structure of the model.
37

---

Published in Transactions on Machine Learning Research (12/2024)
this
50 frustrating everything
energetic
movie was
frustrating everything seemed
energetic
and
was
totally prepared to have good
time
at
and was
tota
prepared to
have a
good time
least
thought be able stand it
but
wrong
least thought
stand
wrong
first
the
weird looping
it
was
like
watching
america
first
the weird looping
it
was
like
watching
america
S
videos
the damn parents
hated
niest
home videos
damn parents
hated
them
50 much
the Stereo
typical latino Family
them 50
much
the stereo
typical
latino family ?
need
speak with the person responsbble
this
we
need 10
speak with the person responsible for this
we
need
have
talk
that little girl who was always
need
have talk
that littlc girl
who was a awways
hanging
on somcone
just
hated her and had to mention hanging
on someone
just hated her and had
mention
now
the final scene
transc ends must say
now
the final scene transc ends
must say
50
glor yously
and full
bad ness that
5| glor jously
bad
and
full
bad
ness that is
movie
its
own
what
crappy dancing
homrile and
movie
what
crappy dancing horrible
and
beautiful
at once
beautiful
once
(a) GPT-4-turbo
(b) GPT-4o-mini
Figure 13: We apply SLALOM to OpenAT GPT-4-turbo and GPT-4o to showcase its scalability. We use SLALOM-eff scores and linearize them to obtain the highlighed attributions.
F.8
Case study: Identifying Vulnerabilites and Weaknesses in classification models with SLALOM
In our public code repository, we provide case study to highlight how SLALOM can be used to uncover practical flaws in transformer classification models. We use a trained BERT model on IMDB sentiment classification as an example and compute SLALOM scores for the tokens the first. 15 samples of the test set (= 1200 tokens). SLALOM's scores make easy to see which tokens have a potentially severe impact on the movie review outcome. We first select all tokens with importance score > 2, all other words have substantial impact, in particular when added to larger sequence. We provide visualization of these tokens in Figure 14. We then make some interesting discoveries mainly based value scores:
There are more words that are negatively interpreted by the model than positive words
2. Out of the words that have highly negative value scores (+importance >2), we identify several words that are some that are not directly negatively connotated, e.g., "anyway", "somehow", "never", "anymore", "probably", "doesn", "maybe", "without", "however", "surprised"
We then show that by a few (4) minor modification steps, e.g., adding some of these words to a review, we can change the classification decision for a review from positive to negative, without essentially altering its content (i.e., we manually construct an adversarial example.) This highlights how SLALOM can intuitively help to uncover 1) the influential tokens that contribute most the decision and 2) allow for fine-grained analysis that can help uncover spurious concepts and give practitioners an intuitive understanding of model's weaknesses.
38

---

Published in Transactions on Machine Learning Research (12/2024)
5.5
5.0
voon
Imporance Score
DONC
3.5
3.0
JUOTINC
Lighteer
2.5
unpendin
Jonice
2.0
juad
Jay
value score
Figure 14: High-impact tokens for BERT identified in our study. Use zoom for best view.
39


Reviews:
[{"review_id": "PfPd8tJmaW", "reviewer": "Reviewer_WXJv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n\n- Well written paper\n- Comparisons are made to common other techniques, (SHAP, LIME, IG, LRP)\n- Even runtime times are reported\n\n- Theoretically everything seems solid, with great presentation and consistent notation.\n\n**Weaknesses**\n- Experimentation is a bit limited, with not a lot of models considered, but given the theoretical demonstrations, I dont think this is too important, if the code is provided.\n\n**Questions**\n- Since you apply SLALOM also to larger models (BLOOM, Mamba), why are these results not included in the main text?\n- How can SLALOM theoretically be fit to SSM models, since they are not transformers?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces SLALOM, a novel surrogate explanation model designed to address the limitations of Transformers in representing linear and generalized additive models. The authors theoretically and empirically demonstrate the shortcomings of traditional linear attribution methods, such as LIME and SHAP, in aligning with the architecture of Transformers. SLALOM assigns attribution scores by modeling token importance and relevance for each token in an input, converting these into linear attributions comparable to existing methods. Through the Softmax-Linked Additive Log-Odds framework, SLALOM faithfully captures Transformer behaviors and provides explanations aligned with human token annotations. Extensive experiments on synthetic and sentiment classification datasets show that SLALOM outperforms traditional methods in both interpretability and alignment with human annotations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please provide the code.", "broader_impact_concerns": "no.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gNhVeVRARu", "reviewer": "Reviewer_NUN1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. The paper is well-written, with a clear and logical flow that makes it easy to follow.\n2. The paper has identified the limitations of traditional XAI methods in transformers and present SLALOM to tackle this issue.\n3. The evaluation results are convincing, providing the empirical evidence to support SLALOM’s effectiveness.\n\n## Weaknesses\n1. The presentation of the major motivation (additivity) fails to convince me. This is because the additivity in Shapley value theory is based on the marginal contributions. For example, in atomic-game in Shapley value theory, regarding $F(t_1, t_2, \\cdots, t_k) = \\psi_{t_1} + \\psi_{t_2} + \\cdots + \\psi_{t_k}$ where $\\psi_{t_i}$ represents the importance of $t_i$, the weights $\\psi_{t_i}$ are computed by using the marginal contributions of $t_i$ all coalitions. The motivation, as argued, does not hold theoretically. \n2. The paper does not justify the properties in Section 5, which play a crucial role in the paper.\n3. Minor concerns: Some figures are not readable and look confusing, for example Figure 3 and Figure 4. I'll give details in the requested revisions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel parameterized surrogate model: **SLALOM** (Softmax-Linked Additive Log-Odds Model), aimed at improving interpretability for transformer architectures. I appreciate the insights offered in this work and find the idea of SLALOM to be a promising approach. I believe the XAI community would be benefited from this research.\n\nThe authors argue that traditional feature attribution methods like SHAP and LIME, which rely on the additivity assumption, fail to accurately represent feature importance in transformers due to the complex, context-dependent interactions in their attention mechanisms. However, this motivation fails to convince me. I will offer revision requests later.\n\nTo tackle the issue in traditional interpretability methods, such as LIME, SLALOM decomposes the prediction scores into two parts: $F(t) = \\sum a_i(t) v(t_i)$ where $a_i(t)$ represents the contextual importance for token $t_i$ and $v(t_i)$ represents the absolute (local) value/importance for token $t_i$, if I understand correctly.\n\n\nI am convinced by the method, even though the motivation is not properly presented. I think this decomposed interpretability is better to capture token interactions within transformers. The paper provides both theoretical analysis and empirical evidence to support SLALOM’s higher fidelity and computational efficiency, particularly in contrast to SHAP and LIME on synthetic and real-world datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.  The paper argues that **additivity** does not yield high fidelity for transformers, where fidelity is crucial for effective explainability. However, in Shapley value theory, **additivity** refers specifically to the marginal contributions of features, not to a simple additive composition of predictions. Further clarification on this distinction is crucial for the soundness of the paper’s argument. \n\t - (R1) **I request the authors to re-eaxmine the motivation and their claims in the motivation presentation.** \n\t- (R2) Also, perhaps the left subfigure of **Figure 1** needed to be revised accordingly. \n\n2. In Section 3.1:\n\t- (R3) regarding the \"**Input and output representations**\", the index for the $\\mathcal{V}$ should be $i=0,1,\\cdots,|\\mathcal{V}|$ if I understand correctly. However the manuscript uses $|t|$, which is the number of the input tokens not the length of the vocabulary size.\n\n3. In Section 3.1:\n\t- (R4) why do we define $E=[e_i]$ as matrix? Will we use it later? If not, I would like to suggest the authors to revise or simply remove it. \n\n4.  In Sections 3.1 and 3.2:\n\t- (R5) The manuscript seems to confuse the task head for a two-class classification task with the next-token prediction task, which involves predicting across the entire vocabulary. Could you clarify this distinction?\n\n5.  Figure 2:\n\t- (R6) Figure 2 is somewhat confusing. If the authors intend to illustrate the overall structure of a transformer with multiple layers, it would be helpful to annotate with “\\times N” to indicate stacking with N layers, as is common in previous literature. Alternatively, if Figure 2 refers to a single layer only, please clarify this.\n\n6. Section 3.3?\n\t- (R7) This section seems not to fit in the reading flow. It appears a bit random to me.\n\n7. In Section 4.1:\n\t- (R8) Could you clarify what is \"**The transformer .... identical tokens but different length**\". Is a complete sentence? I found this sentence may not be a complete sentence.\n\n8. In section 4.1, \n\t- (R9) we can not conclude that the weights in Shapley value or attribution analysis are the attention scores. So proposition 4.1 may not hold.\n\n9. In section 4.1, \n\t- (R10) can the authors clarify the notation $\\stackrel{!}{=}$?\n\n10. Section 4.1,\n\t- (R11) The paragraph in section 4.1 from \"The transformer gives rise to ...\" is dense and not easy to follow.  I've read several times to understand them. I would like to suggest to polish.\n\n11. Section 4.1,\n\t- (R12) The conclusion in Section 4.1 doesn't hold since the Positional Encoding. In Transformers, when we add positional embeddings the conclusion doesn't hold. Consider, $p_i$ represents the positional encoding at position $I$, the embedding for token $e_i$, the function $g(e_i, p_i)$ is to encode positional information into the embedding. The $g$ must satisfy some good property (*eg* non-symmetry, *etc*). In this case, the conclusion does not hold. Please clarify.\n\n12. Terms.\n\t- (R13) Could you please use **map** instead of **mapping** to follow the convention/practice in mathematics?\n\n13. Section 5.\n\t- (R14) So the work doesn't satisfy the axioms in Shapley value theory. Can you clarify?\n\n14.  Section 5\n        - (R15) Can the author formulate how do you the optimize/solve the $v(\\tau)$ and $s(\\tau)$?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dM3VjwRo7x", "reviewer": "Reviewer_qMY8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\nThe introduction of SLALOM offers a good approach to model interpretability within the transformer framework. By addressing a known issue with traditional attribution methods, this model can push forward the applicability of explainability in complex domains where transformers are widely used.\n\nThe authors provide rigorous proofs demonstrating the fundamental limitations of using linear or additive models as surrogate explanations for transformers. These proofs add a strong theoretical backing to the study, differentiating it from more empirical studies.\n\nExperiments are performed on synthetic and real-world datasets to show the efficacy of SLALOM, especially its efficiency in high-fidelity explanations compared to other methods like LIME and SHAP.\n\n\n**Weaknesses:**\n\nSLALOM operates at the token level, assigning individual importance and value scores. However, it lacks mechanisms for contextual or higher-order interpretability, which are crucial in real-world NLP tasks where the meaning and impact of phrases, clauses, or sentences as a whole are often more significant than isolated tokens. \n\n\nIn transformer models, tokens can sometimes gain high importance due to dataset-specific correlations that do not generalize well. SLALOM’s reliance on token-level importance scores might inadvertently amplify these spurious correlations without a mechanism to account for them.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary:**\n\nThis paper addresses the limitations of traditional feature attribution methods in explaining the output of transformer models, which are widely used in natural language processing and other fields. Specifically, it demonstrates the incompatibility between transformer architectures and linear or additive surrogate models, highlighting that transformers are structurally unable to align with these traditional models. \n\n**Contributions:**\n\nThe authors introduce the Softmax-Linked Additive Log-Odds Model (SLALOM) as a new surrogate model. SLALOM is designed to work with the transformer framework, aiming to provide more accurate and computationally efficient explanations. The authors claim that SLALOM provides higher fidelity explanations with less computational cost than existing methods and validate this claim through synthetic and real-world datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "For a more holistic interpretability approach, provide examples or scenarios where SLALOM’s two-dimensional token representation enhances practical understanding for non-technical stakeholders. For instance, a case study or visualization showing how SLALOM can help end-users interpret a model's decision would be valuable (not in the main text, in the public repo, or appendix).\n\nThe paper mentions that SLALOM performs efficiently on smaller models but lacks detail on scalability in large transformer-based models, which are often deployed in real-world applications. Testing SLALOM on models larger than 7B parameters (if feasible) or discussing potential scaling strategies could be insightful.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents SLALOM, a surrogate explanation model designed to explain the outputs of transformers. It addresses the limitations of earlier linear or additive surrogate models like LIME and SHAP, which struggle with transformer architectures due to their attention mechanism. SLALOM tackles this by decomposing attribution scores into two components: token importance and relevance scores for each input token. The paper provides theoretical justification and empirical results. Experiments on synthetic and sentiment classification datasets demonstrate that SLALOM outperforms previous methods.

All reviewers commend the paper's clear writing and strong empirical performance. Reviewer WXJv and qMY8 also emphasize the extensive theoretical analysis provided. However, some concerns were raised. Reviewer WXJv suggests that larger models results be included in the main paper rather than appendix. Reviewer qMY8 raises a concern that relying on token-level importance might limit the method's real-world applicability. Reviewer NUN1 points out issues with the motivation and presentation clarity in Section 5, and Figures 3 and 4.

During the rebuttal process, the authors clarified some misleading points in their writing and provided additional experimental results demonstrating the algorithm's broader applicability. After reviewing the paper, comments, and rebuttals, the action editor believes that most concerns have been addressed in rebuttal and recommends accepting the paper. The authors are advised to revise the paper to improve the writing clarity and include discussions from the rebuttal in the final version.
Decision: Accept (poster)