DecompX: Explaining Transformers Decisions by Propagating Token Decomposition
Year: 2023

Abstract:
An emerging solution for explaining Transformer-based models is to use vectorbased analysis on how the representations are formed. However, providing a faithful vector-based explanation for a multi-layer model could be challenging in three aspects: (1) Incorporating all components into the analysis, (2) Aggregating the layer dynamics to determine the information flow and mixture throughout the entire model, and (3) Identifying the connection between the vector-based analysis and the model’s predictions. In this paper, we present DecompX to tackle these challenges. DecompX is based on the construction of decomposed token representations and their successive propagation throughout the model without mixing them in between layers. Additionally, our proposal provides multiple advantages over existing solutions for its inclusion of all encoder components (especially nonlinear feed-forward networks) and the classification head. The former allows acquiring precise vectors while the latter transforms the decomposition into meaningful prediction-based values, eliminating the need for normor summation-based vector aggregation. According to the standard faithfulness evaluations, DecompX consistently outperforms existing gradient-based and vector-based approaches on various datasets. Our code is available at github.com/mohsenfayyaz/DecompX.

heading: 1 Introduction
text: While Transformer-based models have demonstrated significant performance, their black-box nature necessitates the development of explanation methods for understanding these models’ decisions (Serrano and Smith, 2019; Bastings and Filippova, 2020; Lyu et al., 2022). On the one hand, researchers have adapted gradient-based methods
⋆ Equal contribution.
from computer vision to NLP (Li et al., 2016; Wu and Ong, 2021). On the other hand, many have attempted to explain the decisions based on the components inside the Transformers architecture (vector-based methods). Recently, the latter has shown to be more promising than the former in terms of faithfulness (Ferrando et al., 2022).
Therefore, we focus on the vector-based methods which require an accurate estimation of (i) the mixture of tokens in each layer (local-level analysis), and (ii) the flow of attention throughout multiple layers (global-level analysis) (Pascual et al., 2021). Some of the existing local analysis methods include raw attention weights (Clark et al., 2019), effective attentions (Brunner et al., 2020), and vector norms (Kobayashi et al., 2020, 2021), which all attempt to explain how a single layer combines its input representations. Besides, to compute the global impact of the inputs on the outputs, the local behavior of all layers must be aggregated. Attention rollout and attention flow were the initial approaches for recursively aggregating the raw attention maps in each layer (Abnar and Zuidema, 2020). By employing rollout, GlobEnc (Modarressi et al., 2022) and ALTI (Ferrando et al., 2022) significantly improved
2649
on previous work by substituting norm-based local methods (Kobayashi et al., 2021) for raw attentions. Despite their advancements, these vectorbased methods still have three major limitations: (1) they ignore the encoder layer’s Feed-Forward Network (FFN) because of its non-linearities, (2) they use rollout, which produces inaccurate results because it requires scalar local attributions rather than decomposed vectors which causes information loss, and (3) they do not take the classification head into account.
In an attempt to address all three limitations, in this paper, we introduce DecompX. Instead of employing rollout to aggregate local attributions, DecompX propagates the locally decomposed vectors throughout the layers to build a global decomposition. Since decomposition vectors propagate along the same path as the original representations, they accurately represent the inner workings of the entire model. Furthermore, we incorporate the FFNs into the analysis by proposing a solution for the non-linearities. The FFN workaround, as well as the decomposition, enable us to also propagate through the classification head, yielding per predicted label explanations. Unlike existing techniques that provide absolute importance, this per-label explanation indicates the extent to which each individual token has contributed towards or against a specific label prediction (Figure 1).
We conduct a comprehensive faithfulness evaluation over various datasets and models, that verifies how the novel aspects of our methodology contribute to more accurate explanations. Ultimately, our results demonstrate that DecompX consistently outperforms existing well-known gradientand vector-based methods by a significant margin.

heading: 2 Related Work
text: Vector-based analysis has been sparked by the motivation that attention weights alone are insufficient and misleading to explain the model’s decisions (Serrano and Smith, 2019; Jain and Wallace, 2019). One limitation was that it neglects the selfattention value vectors multiplied by the attention weights. Kobayashi et al. (2020) addressed it by using the norm of the weighted value vectors as a measure of inter-token attribution. Their work could be regarded as one of the first attempts at Transformer decomposition. They expanded their analysis from the self-attention layer to the entire attention block and found that residual connections
are crucial to the information flow in the encoder layer (Kobayashi et al., 2021).
However, to be able to explain the multilayer dynamics, one needs to aggregate the local analysis into global by considering the attribution mixture across layers. Abnar and Zuidema (2020) introduce the attention rollout and flow methods, which aggregate multilayer attention weights to create an overall attribution map. Nevertheless, the method did not result in accurate maps as it was based on an aggregation of attention weights only. GlobEnc (Modarressi et al., 2022) and ALTI (Ferrando et al., 2022) improved this by incorporating decomposition at the local level and then aggregating the resulting vectors-norms with rollout to build global level explanations. At the local level, GlobEnc extended Kobayashi et al. (2021) by incorporating the second Residual connection and LayerNormalization layer after the attention block. GlobEnc utilizes the L2-norm of the decomposed vectors as an attribution measure; however, Ferrando et al. (2022) demonstrate that the reduced anisotropy of the local decomposition makes L2-norms an unreliable metric. Accordingly, they develop a scoring metric based on the L1-distances between the decomposed vectors and the output of the attention block. The final outcome after applying rollout, referred to as ALTI, showed improvements in both the attention-based and norm-based scores.
Despite continuous improvement, all these methods suffer from three main shortcomings. They all omitted the classification head, which plays a significant role in the output of the model. In addition, they only evaluate linear components for their decomposition, despite the fact that the FFN plays a significant role in the operation of the model (Geva et al., 2021, 2022). Nonetheless, the most important weakness in their analysis is the use of rollout for multi-layer aggregation.
Rollout assumes that the only required information for computing the global flow is a set of scalar cross-token attributions. Nevertheless, this simplifying assumption ignores that each decomposed vector represents the multi-dimensional impact of its inputs. Therefore, losing information is inevitable when reducing these complex vectors into one cross-token weight. On the contrary, by keeping and propagating the decomposed vectors in DecompX, any transformation applied to the representations can be traced back to the input tokens without information loss.
Gradient-based methods. One might consider gradient-based explanation methods as a workaround to the three issues stated above. Methods such as vanilla gradients (Simonyan et al., 2014), GradientXInput (Kindermans et al., 2016), and Integrated gradients (Sundararajan et al., 2017) all rely on the gradients of the prediction score of the model w.r.t. the input embeddings. To convert the gradient vectors into scalar per-token importance, various reduction methods such as L1-norm (Li et al., 2016), L2-norm (Poerner et al., 2018), and mean (Atanasova et al., 2020; Pezeshkpour et al., 2022) have been employed. Nonetheless, Bastings et al. (2022) evaluations showed that none of them is consistently better than the other. Furthermore, adversarial analysis and sanity checks both have raised doubts about gradient-based methods’ trustworthiness (Wang et al., 2020; Adebayo et al., 2018; Kindermans et al., 2019).
Perturbation-based methods. Another set of interpretability methods, broadly classified as perturbation-based methods, encompasses widely recognized approaches such as LIME (Ribeiro et al., 2016) and SHAP (Shapley, 1953). However, these were excluded from our choice of comparison techniques, primarily due to their documented inefficiencies and reliability issues as highlighted by Atanasova et al. (2020). We follow recent work (Ferrando et al., 2022; Mohebbi et al., 2023) and mainly compare against gradient-based methods which have consistently proven to be more faithful than perturbation-based methods.
Mohebbi et al. (2023) recently presented a method called Value zeroing to measure the extent of context mixing in encoder layers. Their approach involves setting the value representation of each token to zero in each layer and then calculating attribution scores by comparing the cosine distances with the original representations. Although they focused on local-level faithfulness, their global experiment has clear drawbacks due to its reliance on rollout aggregation and naive evaluation metric (cf. A.3).

heading: 3 Methodology
text: Based on the vector-based approaches of Kobayashi et al. (2021) and Modarressi et al. (2022), we propose decomposing token representations into their constituent vectors. Consider decomposing the ith token representation in layer ℓ ∈ {0, 1, 2, ..., L, L + 1}1, i.e., xℓi ∈ {xℓ1,xℓ2, ...,xℓN}, into elemental vectors attributable to each of the N input tokens:
xℓi = N∑
k=1
xℓi⇐k (1)
According to this decomposition, we can compute the norm of the attribution vector of the kth input (xℓi⇐k) to quantify its total attribution to x ℓ i . The main challenge of this decomposition, however, is how we could obtain the attribution vectors in accordance with the internal dynamics of the model.
1ℓ = 0 is the input embedding layer and ℓ = L+ 1 is the classification head over the last encoder layer.
As shown in Figure 2, in the first encoder layer, the first set of decomposed attribution vectors can be computed as x2i⇐k.
2 These vectors are passed through each layer in order to return the decomposition up to that layer: xℓi⇐k → Encoderℓ → xℓ+1i⇐k. Ultimately, the decomposed vectors of the [CLS] token are passed through the classification head, which returns a decomposed set of logits. These values reveal the extent to which each token has influenced the corresponding output logit.
In this section, we explain how vectors are decomposed and propagated through each component, altogether describing a complete propagation through an encoder layer. After this operation is repeated across all layers, we describe how the classification head transforms the decomposition vectors from the last encoder layer into prediction explanation scores.

heading: 3.1 The Multi-head Self-Attention
text: The first component in each encoder layer is the multi-head self-attention mechanism. Each head, h ∈ {1, 2, ...,H}, computes a set of attention weights where each weight αhi,j specifies the raw attention from the ith to the jth token. According to Kobayashi et al. (2021)’s reformulation, the output of multi-head self-attention, zℓi , can be viewed as the sum of the projected value transformation (vh(x) = xW hv + b h v ) of the input over all heads:
zℓi =
H∑
h=1
N∑
j=1
αhi,jv h(xℓj)W h O + bO (2)
The multi-head mixing weight W hO and bias bO could be combined with the value transformation to form an equivalent weight W hAtt and bias bAtt in a simplified format3:
zℓi = H∑
h=1
N∑
j=1
αhi,jx ℓ jW h Att︸ ︷︷ ︸
zℓi←j
+bAtt (3)
Since Kobayashi et al. (2021) and Modarressi et al. (2022) both use local-level decomposition, they regard zℓi←j as the attribution vector of token i from input token j in layer ℓ’s multi-head attention.4 We also utilize this attribution vector, but only in the first encoder layer since its inputs are also the same
2As x denotes the inputs, the output decomposition of the first layer is the input of the second layer.
3cf. A.1 for further detail on the simplification process. 4Note that even though they discard the bias within the
head-mixing module, bO , the value bias bhv is included.
inputs of the whole model (z1i←j = z 1 i⇐j). For other layers, however, each layer’s decomposition should be based on the decomposition of the previous encoder layer. Therefore, we plug Eq. 1 into the formula above:
zℓi = H∑
h=1
N∑
j=1
αhi,j
N∑
k=1
xℓj⇐kW h Att + bAtt
= N∑
k=1
H∑
h=1
N∑
j=1
αhi,jx ℓ j⇐kW h Att + bAtt
(4)
To finalize the decomposition we need to handle the bias which is outside the model inputs summation ( ∑N k=1). One possible workaround would be to simply omit the model’s internal biases inside the self-attention layers and other components such as feed-forward networks. We refer to this solution as NoBias. However, without the biases, the input summation would be incomplete and cannot recompose the inner representations of the model. Also, if the decomposition is carried out all the way to the classifier’s output without considering the biases, the resulting values will not tally up to the logits predicted by the model. To this end, we also introduce a decomposition method for the bias vectors with AbsDot, which is based on the absolute value of the dot product of the summation term (highlighted in Eq. 4) and the bias:
ωk = |bAtt · zℓi⇐k,[NoBias]|∑N k=1 |bAtt · zℓi⇐k,[NoBias]|
(5)
where ωk is the weight that decomposes the bias and enables it to be inside the input summation:
zℓi = N∑
k=1
( H∑
h=1
N∑
j=1
αhi,jx ℓ j⇐kW h Att + ωkbAtt)
︸ ︷︷ ︸ zℓi⇐k
(6)
The rationale behind AbsDot is that the bias is ultimately added into all vectors at each level; consequently, the most affected decomposed vectors are the ones that have the greatest degree of alignment (in terms of cosine similarity) and also have larger norms. The sole usage of cosine similarity could be one solution but in that case, a decomposed vector lacking a norm (such as padding tokens) could also be affected by the bias vector. Although alternative techniques may be employed, our preliminary
quantitative findings suggested that AbsDot represents a justifiable and suitable selection.
Our main goal from now on is to try to make the model inputs summation ∑N k=1 the most outer sum, so that the summation term (zℓi⇐k for the formula above) ends up as the desired decomposition.5

heading: 3.2 Finalizing the Attention Module
text: After the multi-head attention, a residual connection adds the layer’s inputs (xℓi) to z ℓ i , producing the inputs of the first LayerNormalization (LN#1):
z̃ℓi = LN(z +ℓ i)
= LN(xℓi + N∑
k=1
zℓi⇐k)
= LN(
N∑
k=1
[xℓi⇐k + z ℓ i⇐k])
(7)
Again, to expand the decomposition over the LN function, we employ a technique introduced by Kobayashi et al. (2021) in which the LN function is broken down into a summation of a new function g(.):
LN(z+ ℓ i) =
N∑
k=1
g z+ ℓ i (z+ ℓ i⇐k) + β ︸ ︷︷ ︸ z̃ℓi⇐k
g z+ ℓ i (z+ ℓ i⇐k) :=
z+ ℓ i⇐k −m(z+ ℓ i⇐k)
s(z+ ℓ i)
⊙ γ
(8)
where m(.) and s(.) represent the input vector’s element-wise mean and standard deviation, respectively.6 Unlike Kobayashi et al. (2021) and Modarressi et al. (2022), we also include the LN bias (β) using our bias decomposition method.

heading: 3.3 Feed-Forward Networks Decomposition
text: Following the attention module, the outputs enter a 2-layer Feed-Forward Network (FFN) with a nonlinear activation function (fact):
zℓFFN = FFN(z̃ ℓ i)
= fact(z̃ ℓ iW 1 FFN + b 1 FFN︸ ︷︷ ︸
ζℓi
)W 2FFN + b 2 FFN (9)
5For a bias-included analysis, note that the bias weighting in all subsequent decomposition equations is always determined by the bias itself and its prior term (highlighted in the above formula).
6γ ∈ Rd and β ∈ Rd are respectively the trainable scaling and bias weights of LN. For extra details, please refer to Appendix A in Kobayashi et al. (2021) for the derivation.
W λFFN and b λ FFN represent the weights and biases, respectively, with λ indicating the corresponding layer within the FFN. In this formulation, the activation function is the primary inhibiting factor to continuing the decomposition. As a workaround, we approximate and decompose the activation function based on two assumptions: the activation function (1) passes through the origin (fact(0) = 0) and (2) is monotonic.7 The approximate function is simply a zero intercept line with a slope equal to the activation function’s output divided by its input in an elementwise manner:
f (x) act (x) = θ (x) ⊙ x
θ(x) := (θ1, θ2, ...θd) s.t. θt = fact(x
(t))
x(t)
(10)
where (t) denotes the dimension of the corresponding vector. One important benefit of this alternative function is that when x is used as an input, the output is identical to that of the original activation function. Hence, the sum of the decomposition vectors would still produce an accurate result. Using the described technique we continue our progress from Eq. 9 by decomposing the activation function:
zℓFFN,i = f (ζℓi) act (
N∑
k=1
ζℓi⇐k)W 2 FFN + b 2 FFN
= ∑
k=1
θ(ζ ℓ i) ⊙ ζℓi⇐k + b2FFN︸ ︷︷ ︸
zℓFFN,i⇐k
(11)
In designing this activation function approximation, we prioritized completeness and efficiency. For the former, we ensure that the sum of decomposed vectors should be equal to the token’s representation, which has been fulfilled by applying the same θ to all decomposed values ζ based on the line passing the activation point. While more complex methods (such as applying different θ to each ζ) which require more thorough justification may be able to capture the nuances of different activation functions more accurately, we believe that our approach strikes a good balance between simplicity and effectiveness, as supported by our empirical results.
The final steps to complete the encoder layer progress are to include the other residual connection and LayerNormalization (LN#2), which could be handled similarly to Eqs. 7 and 8:
7Even though the GeLU activation function, which is commonly used in BERT-based models, is not a monotonic function in its x < 0 region, we ignore it since the values are small.
xℓ+1i = LN( N∑
k=1
[z̃ℓi⇐k + z ℓ FFN,i⇐k︸ ︷︷ ︸
zℓ FFN+,i⇐k
])
= N∑
k=1
gzℓ FFN+,i
(zℓ FFN+,i⇐k) + β
︸ ︷︷ ︸ xℓ+1i⇐k
(12)
Using the formulations described in this section, we can now obtain xℓ+1i⇐k from x ℓ i⇐k, and by continuing this process across all layers, xL+1i⇐k is ultimately determined.

heading: 3.4 Classification Head
text: Norm- or summation-based vector aggregation could be utilized to convert the decomposition vectors into interpretable attribution scores. However, in this case, the resulting values would only become the attribution of the output token to the input token, without taking into account the taskspecific classification head. This is not a suitable representation of the model’s decision-making, as any changes to the classification head would have no effect on the vector aggregated attribution scores. Unlike previous vector-based methods, we can include the classification head in our analysis thanks to the decomposition propagation described above.8 As the classification head is also an FFN whose final output representation is the prediction scores y = (y1, y2, ..., yC) for each class c ∈ {1, 2, ..., C}, we can continue decomposing through this head as well. In general, the [CLS] token representation of the last encoder layer serves as the input for the two-layer (pooler layer + classification layer) classification head:
y = uact(x L+1 [CLS]W pool+bpool)W cls+bcls (13)
Following the same procedure as in Section 3.3, we can now compute the input-based decomposed vectors of the classification head’s output yk using the decomposition of the [CLS] token, xi⇐k. By applying this, in each class we would have an array of attribution scores for each input token, the sum of which would be equal to the prediction score of the model for that class:
yc =
N∑
k=1
yc⇐k (14)
To explain a predicted output, yc⇐k would be the attribution of the kth token to the total prediction score.
8We also discuss about alternative use cases in section A.2

heading: 4 Experiments
text: Our faithfulness evaluations are conducted on four datasets covering different tasks, SST-2 (Socher et al., 2013) for sentiment analysis, MNLI (Williams et al., 2018) for NLI, QNLI (Rajpurkar et al., 2016) for question answering, and HateXplain (Mathew et al., 2021) for hate speech detection. Our code is implemented based on HuggingFace’s Transformers library (Wolf et al., 2020). For our experiments, we used fine-tuned BERT-baseuncased (Devlin et al., 2019) and RoBERTa-base (Liu et al., 2019), obtained from the same library.9 As for gradient-based methods, we choose 0.1 as a step size in integrated gradient experiments and consider the L2-Norm of the token’s gradient vector as its final attribution score.10

heading: 4.1 Evaluation Metrics
text: We aim to evaluate our method’s Faithfulness by perturbing the input tokens based on our explanations. A widely-used perturbation method removes K% of tokens with the highest / lowest estimated importance to see its impact on the output of the model (Chen et al., 2020; Nguyen, 2018). To mitigate the consequences of perturbed input becoming out-of-distribution (OOD) for the model, we replace the tokens with [MASK] instead of removing them altogether (DeYoung et al., 2020). This approach makes the sentences similar to the pretraining data in masked language modeling. We opted for three metrics: AOPC (Samek et al., 2016), Accuracy (Atanasova et al., 2020), and Prediction Performance (Jain et al., 2020).
AOPC: Given the input sentence xi, the perturbed input x̃(K)i is constructed by masking K% of the most/least important tokens from xi. Afterward, AOPC computes the average change in the predicted class probability over all test data as follows:
AOPC(K) = 1
N
N∑
i=1
p(ŷ | xi)−p(ŷ | x̃(K)i ) (15)
where N is the number of examples, and p(ŷ | .) is the probability of the predicted class. When masking the most important tokens, a higher AOPC is better, and vice versa.
9RoBERTa results can be found in section A.3. 10All were conducted on an RTX A6000 24GB machine.
Accuracy: Accuracy is calculated by averaging the performance of the model over different masking ratios. In cases where tokens are masked in decreasing importance order, lower Accuracy is better, and vice versa.
Predictive Performance: Jain et al. (2020) employ predictive performance to assess faithfulness by evaluating the sufficiency of their extracted rationales. The concept of sufficiency evaluates a rationale—a discretized version of soft explanation scores—to see if it adequately indicates the predicted label (Jacovi et al., 2018; Yu et al., 2019). Based on this, a BERT-based model is trained and evaluated based on inputs from rationales only to see how it performs compared with the original model. As mentioned by Jain et al. (2020), for each example, we select the top-K% tokens based on the explanation methods’ scores to extract a rationale11.
11We select the top 20% for the single sentence and top 40% for the dual sentence tasks.

heading: 4.2 Results
text: Figure 3 demonstrates the AOPC and Accuracy of the fine-tuned model on the perturbed inputs at different corruption rates K. As we remove the most important tokens in this experiment, higher changes in the probability of the predicted class computed by AOPC and lower accuracies are better. Our method outperforms comparison explanation methods, both vector- and gradient-based, by a large margin at every corruption rate on the SST2 dataset. Table 1 shows the aggregated AOPC and Accuracy over corruption rates, as well as Predicted Performance on different datasets. DecompX consistently outperforms other methods, which confirms that a holistic vector-based approach can present higher-quality explanations. Additionally, we repeated this experiment by removing the least important tokens. Figure A.2 and Table A.2 in the Appendix demonstrate that even with 10%-20% of the tokens selected by DecompX the task still performs incredibly well. When keeping only 10% of the tokens based on DecompX, the accuracy only
drops by 2.64% (from 92.89% of the full sentence), whereas the next best vector- and gradient-based methods suffer from the respective drops of 7.34% and 15.6%. In what follows we elaborate on the reasons behind this superior performance.
The role of feed-forward networks. Each Transformers encoder layer includes a feed-forward layer. Modarressi et al. (2022) omitted the influence of FFN when applying decomposition inside each layer due to FFN being a non-linear component. In contrast, we incorporated FFN’s effect by a point-wise approximation (cf. §3.3). To examine its individual effect we implemented GlobEnc + FFN where we incorporated the FFN component in each layer. Table 1 shows that this change improves GlobEnc in terms of faithfulness, bringing it closer to gradient-based methods. Moreover, we conducted a leave-one-out ablation analysis12 to ensure FFN’s effect on DecompX. Figure 4 reveals that removing FFN significantly decreases the AOPC.
The role of biases. Even though Figure 4 demonstrates that considering bias in the analysis only has a slight effect, it is important to add biases for the human interpretability of DecompX. Figure 6 shows the explanations generated for an instance from MNLI by different methods. While the order of importance is the same in DecompX and DecompX W/O Bias, it is clear that adding the bias fixes the origin and describes which tokens had positive (green) or negative (red) effect on the predicted label probability. Another point is that without considering the biases, presumably
12In all our ablation studies, we use norm-based aggregation when not incorporating the classification head: ∥xL+1[CLS]⇐k∥
less influential special tokens such as [SEP] are weighed disproportionately which is corrected in DecompX.13
The role of classification head. Figure 4 illustrates the effect of incorporating the classification head by removing it from DecompX. AOPC drastically drops when we do not consider the classification head, even more than neglecting bias and FFN, highlighting the important role played by the classification head. Moreover, incorporating the classification head allows us to acquire the exact effect of individual input tokens on each specific output class. An example of this was shown earlier in Figure 1, where the explanations are for the predicted class (Positive) in SST2. Figure 6 provides another example, for an instance from the MNLI dataset. Due to their omitting of the classification head, previous vector-based methods assign importance to some tokens (such as “or bolted”) which are actually not important for the predicted label. This is due to the fact that the tokens were important for another label (contradiction; cf. Figure A.1). Importantly, previous methods fall short of capturing this per-label distinction. Consequently, we believe that no explanation method that omits the classification head can be deemed complete.
The role of decomposition. In order to demonstrate the role of propagating the decomposed vectors instead of aggregating them in each layer using rollout, we try to close the gap between DecompX and GlobEnc by simplifying DecompX and incorporating FFN in GlobEnc. With this simplification,
13The importance of special tokens does not change our results as it is not possible to remove the special tokens in the perturbed input.
the difference between DecompX W/O classification head and GlobEnc with FFN setups is that the former propagates the decomposition of vectors while the latter uses norm-based aggregation and rollout between layers. Figure 5 illustrates the clear positive impact of our decomposition. We show that even without the FFN and bias, decomposition can outperform the rollout-based GlobEnc. These results demonstrate that aggregation in-between layers causes information loss and the final attributions are susceptible to this simplifying assumption.

heading: 5 Conclusions
text: In this work, we introduced DecompX, an explanation method based on propagating decomposed token vectors up to the classification head, which addresses the major issues of the previous vectorbased methods. To achieve this, we incorporated all the encoder layer components including nonlinear functions, propagated the decomposed vectors throughout the whole model instead of aggregating them in-between layers, and for the first time, incorporated the classification head resulting in faithful explanations regarding the exact positive or negative impact of each input token on the output classes. Through extensive experiments, we demonstrated that our method is consistently better than existing vector- and gradient-based methods by a wide margin. Our work can open up a new avenue for explaining model behaviors in various situations. As future work, one can apply the technique to encoder-decoder Transformers, multilingual, and Vision Transformers architectures.
Limitations
DecompX is an explanation method for decomposing output tokens based on input tokens of a Transformer model. Although the theory is applicable to other use cases, since our work is focused on English text classification tasks, extra care and
evaluation experiments may be required to be used safely in other languages and settings. Due to limited resources, evaluation of large language models such as GPT-2 (Radford et al., 2019) and T5 (Raffel et al., 2022) was not viable.

heading: A Appendix
text: A.1 Equivalent Weight and Bias in the Attention Module
zℓi = H∑
h=1
N∑
j=1
αhi,j(x ℓ jW h v + b h v)W h O + bO
=
H∑
h=1
N∑
j=1
αhi,j(x ℓ jW h vW h O + b h vW h O) + bO
= H∑
h=1
N∑
j=1
αhi,jx ℓ j W h vW h O︸ ︷︷ ︸
W hAtt
+ H∑
h=1
bhvW h O
1 N∑
j=1
αhi,j + bO
︸ ︷︷ ︸ bAtt
(16)
A.2 Alternative use cases The versatility of DecompX allows for explaining various NLP tasks and use cases. Since each output representation is decomposed based on the inputs (xL+1i⇐k ), it can be propagated through the taskspecific head. In Question Answering (QA), for instance, there are two heads to identify the beginning and end of the answer span (Devlin et al., 2019). Thanks to the fact that DecompX is applied posthoc and the final predicted span is known (xL+1i=Start and xL+1i=End), we can continue propagation through the heads as described in Section 3.4. In the end, DecompX can indicate the impact of each input
token on the span selection: yStart⇐k ∈ RN & yEnd⇐k ∈ RN .
A.3 RoBERTa Results Figures A.3 and A.4 demonstrate the results of our evaluations over the RoBERTa-base model.
In a contemporaneous work, Mohebbi et al. (2023) introduced the concept of ValueZeroing to incorporate the entire encoder layer and compute context mixing scores in each layer. Our experiments, as shown in Figures A.3 and A.4, demonstrate the poor performance of this technique at global-level. While it’s possible that mismatching configurations14 contributed to this inconsistency, we believe that the main issue lies in their reliance on an oversimplified evaluation measure for their global-level assessments. Their global level evaluation is based on the Spearman’s correlation between the blank-out scores and various attribution methods (see Section 7 in Mohebbi et al. (2023)). The issue with this evaluation is that the blank-out baseline scores were obtained by removing only one token from the input (leave-one-out) and measuring the change in prediction probability, which cannot capture feature interactions (Lyu et al., 2022). For instance, in the sentence “The movie was great and amusing”, independently removing “great” or “amusing” may not change the sentiment, resulting in smaller scores for these words.
14The authors of the study evaluated the models using blimp probing tasks in a prompting format, whereas we fine-tuned our models on SST-2 and MNLI tasks.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract, 1. Intro
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts?

heading: 4. Experiments

heading: 3 B1. Did you cite the creators of artifacts you used?
text: 4. Experiments
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 4. Experiments
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
7 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. The size of the datasets does not affect explanation extraction.
C 3 Did you run computational experiments?

heading: 4. Experiments

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 4. Experiments
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 4. Experiments

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4. Experiments

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4. Experiments
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: As future work, one can apply the technique to encoder-decoder Transformers, multilingual, and Vision Transformers architectures. Limitations
DecompX is an explanation method for decomposing output tokens based on input tokens of a Transformer model. Although the theory is applicable to other use cases, since our work is focused on English text classification tasks, extra care and
evaluation experiments may be required to be used safely in other languages and settings. Due to limited resources, evaluation of large language models such as GPT-2 (Radford et al. , 2019) and T5 (Raffel et al. , 2022) was not viable