Local Interpretation of Transformer Based on Linear Decomposition
Year: 2023

Abstract:
In recent years, deep neural networks (DNNs) have achieved state-of-the-art performance on a wide range of tasks. However, limitations in interpretability have hindered their applications in the real world. This work proposes to interpret neural networks by linear decomposition and finds that the ReLU-activated Transformer can be considered as a linear model on a single input. We further leverage the linearity of the model and propose a linear decomposition of the model output to generate local explanations. Our evaluation of sentiment classification and machine translation shows that our method achieves competitive performance in efficiency and fidelity of explanation. In addition, we demonstrate the potential of our approach in applications with examples of error analysis on multiple tasks.1

heading: 1 Introduction
text: Deep neural networks (DNNs) such as Transformers (Vaswani et al., 2017) have achieved state-of-the-art results on various natural language tasks (Devlin et al., 2019; Liu et al., 2019; Brown et al., 2020; Dai et al., 2019) via learning complex nonlinear relationships of inputs.
However, the lack of interpretability of the predictions given by black-box models limits their application in real-world (Guidotti et al., 2018; Lipton, 2018; Ribeiro et al., 2016b).
A typical way to understand model prediction dynamics is to generate prediction explanations for each input, called local explanation generation (Chen et al., 2020). Most existing works on local explanation algorithms in NLP strive to understand such dynamics on word-level or phraselevel by assigning importance scores on input features (Ribeiro et al., 2016a; Lei et al., 2016; Lundberg et al., 2018; Plumb et al., 2018). However,
∗ Corresponding author. 1We release our algorithm toolkit at https://github.
com/DoubleVII/pydec.
nonlinearity in models makes it difficult to assign the contribution of individual words or phrases to predictions, while the linear counterparts are more interpretable as the weight of each component could be naturally interpreted as its contribution.
In this work, we present a linear decomposition theory to interpret linear models, which can be generalized to nonlinear DNNs. That is, we formalize the decomposition of the linear outputs into components corresponding to the input features, then mathematically propose the properties of linear decomposition and the uniqueness of the decomposition under these properties.
Furthermore, we prove that the ReLU-activated Transformer can be regarded as a linear function by given input features if the causal relationship between the input and certain intermediate variables is disregarded. Therefore, generalize the proposed linear decomposition to Transformer under such an assumption.
However, this decomposition yields a component corresponding to the parameters of additive bias (usually used in the linear layer), which contains a partial contribution of the inputs. Thus we separate and reallocate this part of the contribution to the input features while preserving the mathematical properties of the decomposition.
Quantitative experiments were conducted on sentiment classification and machine translation to identify important input features. We show that our local explanation algorithms efficiently outperform several competitive baselines. Additionally, we propose further implementation of our algorithm to explain the model errors on natural language tasks. The fidelity of our algorithm exceeds that of other baselines.
Our key contributions are summarized as follows:
• We prove the linearity of the ReLU-activated Transformer for a given input under reasonable assumptions.
10270
• We design algorithms for the linear decomposition of Transformer hidden states and propose methods for reallocating the contribution of additive bias while maintaining the mathematical properties.
• Experimental results and case studies on sentiment classification and machine translation validate the fidelity and interpretability of the proposed methods.

heading: 2 Method
text: In this section, we propose the decomposition theory of linear functions. Then, we generalize it to nonlinear cases (i.e., Transformer) and present several decomposition methods accordingly. Finally, we analyze the mathematical properties of the different methods.

heading: 2.1 Linear Decomposition Theory
text: Decomposing the output of a linear system according to its input is relatively simple. The results of the decomposition are intuitively interpreted as the contributions of the inputs to the outputs. We present a theory of linear decomposition, including the definition of decomposition, linear decomposability, and the properties of interpretable decomposition.
Given a set X = {x1, · · · , xm} and a function f , the output is denoted as h = f(X).
Definition 1. (linearly decomposable). The output h of the function f is linearly decomposable for input X if and only if h can be represented as a linear combination of X:
h = f(X) = m∑
i
WXi xi, (1)
where xi ∈ Rn(xi) denotes the i-th input vector, WXi ∈ Rn(h)×n(x) is the linear transformation matrix with respect to xi, and the input X is defined as the basis of the decomposition. Here we use n(·) to denote the dimension of ·.
For linearly decomposable h, it is intuitive to regard WXi xi in Eq. (1) as the contribution of xi. Sometimes input features are divided into different groups, and we are more interested in the overall impact of each group (e.g., tokens split from the same word can be divided into a group to produce word-level explanations). Specifically, a group is an element of a set P , where P is an arbitrary partition of the basis X .
Definition 2. (decomposition). A decomposition of h under partition P is the splitting of h into components corresponding to all groups in P , i.e.,
h = ∑
g∈P
Dh
Dg ∣∣∣∣ P ,
where DhDg ∣∣∣ P
denotes the component corresponding to group g under partition P in the decomposition of h. In this paper, the partition P is omitted if there is no ambiguity.
Considering the given partition P1 = {{x1, x2}, {x3, · · · , xm}} as an example, the decomposition of h under P1 is denoted as
h = Dh
D{x1, x2} ∣∣∣∣ P1 + Dh D{x3, · · · , xm} ∣∣∣∣ P1 .
Since there are exponential decompositions for a function, each with unclear interpretability, we examine the following properties:
Property 1. Orthogonality.
Dxi Dg = { xi, if xi ∈ g 0, otherwise .
Property 2. Linearity.
Dh1 Dg + Dh2 Dg = D(h1 + h2) Dg ,
W Dh
Dg =
D(Wh)
Dg .
Property 3. Group Additivity.
Dh Dg1 + Dh Dg2 =
Dh
Dg1 ∪ g2 .
Definition 3. (interpretable decomposition). A decomposition D is interpretable if it satisfies Orthogonality and Linearity.
The interpretable decomposition specifies the necessary conditions that guarantee interpretability under linear operations. The Group Additivity is related to the consistency of a decomposition.
Definition 4. (consistency). A decomposition D is consistent if DhDg are equal for the same group g in any partition of the basis.
For example, given the partition P1 = {{x1}, · · · , {xm}}, the decomposition of h can be formulated as
h = Dh
D{x1} ∣∣∣∣ P1 + · · ·+ Dh D{xm} ∣∣∣∣ P1 , (2)
and given another partition P2 = {{x1}, {x2, · · · , xm}}, we have
h = Dh
D{x1} ∣∣∣∣ P2 + Dh D{x2, · · · , xm} ∣∣∣∣ P2 . (3)
If D is consistent, then DhD{x1} ∣∣∣ P1 = DhD{x1} ∣∣∣ P2 holds. Consistency guarantees the consistent contribution of a given group by arbitrary partitions from the perspective of interpretability. To determine a consistent decomposition, we propose the following lemma (proved in Appendix A): Lemma 1. A decomposition D is consistent if and only if it satisfies the Group Additivity.
To the best of our knowledge, most of the current local explanation algorithms (Singh et al., 2019; Chen et al., 2020; Li et al., 2016; Sundararajan et al., 2017) are interpretable. Furthermore, for linearly decomposable h, these algorithms are essentially equivalent to the following decomposition: Definition 5. (decomposition D̄). D̄ is defined on linearly decomposable h, where each component D̄h D̄g ∣∣∣ P
is the sum of terms corresponding to the given group g ∈ P , and each term comes from the linear combination of X about h, i.e.,
D̄h
D̄g ∣∣∣∣ P := ∑ xi∈g WXi xi.
D̄ is intuitive, and more importantly, the unique interpretable decomposition for any linearly decomposable h (proved in Appendix B).
Obviously, D̄ satisfies Group Additivity thus is consistent. As forementioned, most existing methods are equivalent and consistent under linear conditions. However, they may lose consistency with nonlinear functions. This aspires us to transform nonlinear functions into locally linear ones for consistency guarantees of the interpretable decomposition, and extend the interpretable decomposition onto nonlinear activation functions while maintaining consistency.

heading: 2.2 ReLU-activated Transformer Is a Linear Function of The Input
text: A typical Transformer (Vaswani et al., 2017) is composed of a stack of identical layers. Each layer of the encoder consists of two major components: a multi-head self-attention mechanism and a feedforward neural network. Besides, a residual connection (He et al., 2016) is employed around each
of the two components, followed by layer normalization (Ba et al., 2016). The decoder is in a similar fashion to the encoder, but with additional attention to draw relevant information from the hidden states generated by the encoders.
Complicated as it may be, a Transformer can be seen as a combination of the above modules. Thus, if each module is linearly decomposable, the final result will be linearly decomposable. To achieve this, we disregard the input’s contribution to the intermediate variables of attention scores and standard deviation of layer normalization. Consequently, these intermediate variables can be considered as coefficients of the linear transformation in the formula, analogous to the parameters of linear layers in the model. Though we partially ignore some of the influence propagations, the remainings retain the major causalities of the model, which are sufficient to provide adequate explanations. We verified this assumption by comparing the performance before and after cutting off the gradient of the attention scores and standard deviations (Section 3.2). To make life easier, this paper assumes the model uses ReLU as the activation function. We discuss the extensibility of our approach to other activation functions in Section 7.
Based on the above elaboration, we give the following lemma, which provides the condition to apply linear decomposition on Transformer.
Lemma 2. For a given input X = {x1, x2, · · · , xm}, any hidden state h in Transformer can be represented as:
h = m∑
i
WXi xi + L∑
l
WBl bl, (4)
where xi denotes the i-th input vector, bl denotes the parameter of additive bias in the model2.
Proof. Proof by mathematical induction. Base Case. For any input xi, we have xi = xi, which is consistent with Eq. (4), i.e.
WXj = { I, if j = i 0, otherwise ,WBl = 0.
Induction step. Assume Eq. (4) holds for all input hidden states of a Transformer sub-layer, it holds for the output of the sub-layer, too. We prove each of the sub-layer types below respectively.
2For ease of expression, all the parameters of additive bias in the model are numbered from 1 to L.
For Linear Layer, we have
h′ = W ′h+ bk
= m∑
i
W ′WXi xi + L∑
l̸=k
W ′WBl bl + (I +W ′WBk )bk.
For Attention Layer, since each attention score ai is considered as a coefficient of the linear transformation, then we have
h′ = a1h1 + · · ·+ amhm
= m∑
j
aj
[ m∑
i
WXij xi + L∑
l
WBlj bl
]
= m∑
i
[ m∑
j
ajW X ij ] xi + L∑
l
[ m∑
j
ajW B lj ] bl.
(5)
For Residual Connection, we have
h′ = h1 + h2
= m∑
i
[ WXi1 +W X i2 ] xi + L∑
l
[ WBl1 +W B l2 ] bl.
As for Layer Normalization, we rewrite a linear transformation
h′ = LN(h) = s(h−W ′h), (6) where the scalar s = 1/ √ V ar(h) is the coefficient and W ′ is the averaging operator, i.e.
W ′ = [1/n(h)]n(h)×n(h).
The Activation Function ReLU can be rewritten as a linear transformation h′ = relu(h) = W ′h, where
W ′ = diag(d1, · · · , dn(h))
di = { 1, if h[i] ≥ 0 0, otherwise . (7)
With Lemma 2, we raise the core theorem of this paper. Theorem 1. For a given input, any hidden state h in Transformer is linearly decomposable on the basis X ′ = {x1, · · · , xm, b1, · · · , bL}.
In other words, we can obtain the decomposition D̄ of h as
h = ∑
g∈P
D̄h D̄g + D̄h D̄B , (8)
where B = {b1, · · · , bL} and we still use P to denote the partition of the input X instead of the basis X ′. The partition of the basis X ′ can be recovered as P ′ = P ∪ {B} if b1, · · · , bL are considered as a single group.
D̄X
+ D̄h
D̄B
. For D̄ (a), D̄relu(h)
D̄B
is gov-
erned by both D̄h D̄B and D̄h D̄X , while for D̂ (b), D̂relu(h) D̂B is only governed by D̂h D̂B and is independent of D̂h D̂X .

heading: 2.3 Decomposing The Contribution of Additive Bias
text: Eq. (8) shows that the parameters of additive bias in the model contribute partially to h, by D̄h
D̄B . This
is reasonable because the term D̄h D̄B represents a prior guess made by the model in the absence of inputs (e.g., even in the absence of inputs, a language model may predict ‘The’ as the beginning of a sentence with a certain probability). However, the term D̄h
D̄B is also mixed with the contribution
from inputs, since the bias component of the ReLU output may change due to the components of the input ( Figure 1 (a)). To address this issue, we define a new decomposition D̂ and require the bias component of the ReLU output to be independent of the input components ( Figure 1 (b)), i.e.,
D̂relu(h)
D̂B := relu(
D̂h D̂B ). (9)
The remaining parts are to be assigned to each group of the input, which is
relu(h)− relu( D̂h D̂B )
=W ′h− relu( D̂h D̂B )
=W ′
 ∑
g∈P
D̂h D̂g + D̂h D̂B
 − relu( D̂h
D̂B )
= ∑
g∈P W ′
D̂h D̂g +
[ W ′ D̂h
D̂B − relu( D̂h D̂B )
] ,
(10)
where W ′ comes from Eq. (7). The first term of Eq. (10) is easily assigned to each group, and the second term implies the contribution separated from the original bias term, which
is split in the assignment:
D̂relu(h)
D̂g := W ′
D̂h D̂g +αg
[ W ′ D̂h
D̂B − relu( D̂h D̂B )
] , (11)
where ∑
g∈P αg = 1. We designed two methods to calculate α.
Absolute-value-based:
αg = | D̂h D̂g | ∑
g∈P | D̂hD̂g | . (12)
Signed-value-based:
αg = D̂h D̂g∑
g∈P D̂h D̂g
. (13)
For linear functions, we introduce Orthogonality and Linearity into D̂ to make it interpretable:
D̂xi
D̂g := { xi, if xi ∈ g 0, otherwise , (14)
D̂(h1 + h2)
D̂g :=
D̂h1
D̂g +
D̂h2
D̂g , (15)
D̂(Wh)
D̂g := W
D̂h D̂g . (16)
Finally, we notice that the α in Eq. (13) explodes as the denominator gets close to 0, degrading the algorithm’s performance. As a comparison, α in Eq. (12) is more stable when constrained by the probability simplex. To alleviate the stability issue, we switch to the absolute-value-based method in the unstable region of Eq. (13). The instability is measured by
r =
∣∣∣ ∑
g∈P D̂h D̂g ∣∣∣ ∑
g∈P ∣∣∣ D̂h D̂g ∣∣∣ , (17)
where r indicates more stability when ascending from 0 to 1. In our experiments, we adopt a hyperparameter λ to interpolate different α schemes: absolute-value-based when r < λ and signedvalue-based when r ≥ λ. When λ goes from 0 to 1, the decomposition D̂ will change from signedvalue-based algorithm to absolute-value-based algorithm with more inconsistency.

heading: 2.4 Comparison
text: Our algorithms exhibit different properties under the two α schemes in Eq. (11), which lead to different final results. The signed-value-based D̂ satisfies Group Additivity, while the absolute-valuebased approach does not satisfy it (Appendix C). More importantly, it can be proved that the signedvalue-based α calculation is the only solution that satisfies Group Additivity (Appendix D), and the absolute-value-based approach aims at the numerical stability issue. By Lemma 1, we conclude that D̂ based on Signed-value is consistent, while the one based on Absolute-value is inconsistent.

heading: 3 Experiments
text: We evaluate our algorithms with SOTA Transformer implementations on text classification (RoBERTa, Liu et al., 2019) and machine translation (Vaswani et al., 2017). It is notable that the classification follows the encoder-only architecture, while the translation follows the encode-decode architecture.

heading: 3.1 Experiment Settings
text: Datasets. We use the SST-2 (Socher et al., 2013) and the IMDB (Maas et al., 2011) datasets for sentiment analysis, which is modeled as a binary classification. The SST-2 includes 6920/872/1821 instances in the train/dev/test sets. The IMDB includes 25000/25000 instances in the train/test sets. We adopt WMT14 English-to-German (En⇒De) for machine translation, with 4.5M parallel sentences consisting of 118M English and 111M German words for training. We use newstest 2013 for validation and newstest 2014 as the test set.
We evaluate the explanation on test sets of all datasets, except for the IMDB, where we test on a subset with 2000 randomly selected samples from test data due to computation expenses.
Models. We adopt the Transformer (Vaswani et al., 2017) base model with baseline settings for machine translation. We adopt the fine-tuned RoBERTa base model (Liu et al., 2019) for text classification. RoBERTa utilizes GELU (Hendrycks and Gimpel, 2016) as its activation function. To apply our decomposition, we replaced it with ReLU during fine-tuning. The impact on performance and other implementation details are explained in Appendix E.
Appendix F shows the best performance of the models on all datasets in our experiments.
Evaluations. We adopt the area over the perturbation curve (AOPC, Chen et al., 2020; Nguyen, 2018; Samek et al., 2016) to evaluate token-level explanations, which measures local fidelity by comparing the probability change on the predicted label after deleting k% top-scored tokens assigned by explanation algorithms. We set k = 20 for sentiment analysis. For machine translation, the number of deleted tokens is fixed at 4. This is because a complete generation consists of multiple token predictions, while each generated target-side token depends on only a few input tokens rather than the entire input sequence. In addition, we average the AOPC scores for the decoding process of the machine translation model.
In this paper, we generate contribution scores by decomposing the logits of the model. Specifically, for a classification of n classes, the model generates a n-dimensional vector of logits ho ∈ Rn for a prediction ŷ = argmaxi h
o[i]. Thus, the importance score of feature xi can be expressed as Dh o
D{xi} [ŷ].

heading: 3.2 Main Results
text: We compare our algorithms with the following baselines: Leave-One-Out (LOO, Li et al., 2016), LIME (Ribeiro et al., 2016b), GlobEnc (Modarressi et al., 2022), Integrated Gradient (IG, Sundararajan et al., 2017), Agglomerative Contextual Decomposition (ACD, Singh et al., 2019), Layer-wise Rel-
evance Propagation (LRP, Voita et al., 2021), and HEDGE (Chen et al., 2020). We also report the AOPC as a reference when random scores are assigned to tokens. For our algorithms, we adopt D̄ and D̂ in the evaluation, and fix the hyperparameter λ of D̂ at 0.1.
As shown in Table 1, the improved decomposition D̂ outperforms our base decomposition D̄ and other baselines in the quality of explanations over the SST-2 dataset, especially the IMDB dataset. Our decomposition D̂ achieves comparable performance to IG on the WMT En-De dataset. IG performs well on the translation but poorly on the sentiment classification with excessive computational complexity. We suspect that this is because the loss scale of the sentiment classification is significantly smaller than that of the translation, weakening the salience of the gradient. Occlusion-based methods,
such as LOO and LIME, achieve relatively good performance on the WMT dataset because they are very similar to the evaluation metrics when k is small. Furthermore, on the IMDB dataset, LOO and LIME become weaker as the sequence becomes longer due to the diminished impact of a single token deletion in a sentence. The ACD fails the IMDB due to accumulated precision error, while the LRP suffers from exponential overhead.
Nevertheless, IG comprehensively considers the influence of each variable, including attention weights and standard deviations of layer normalization. We additionally consider linearizing the attention layer and layer normalization by cutting off the gradients of attention weights and standard deviations for comparison, where we justify our hypothesis of ignoring their influence propagations by looking into its impact on performance. Surprisingly, this hypothesis even gains improvements on the SST-2 and IMDB datasets. To further validate our hypothesis, we investigated the contribution of inputs to outputs through different intermediate variables by examining the norms of the gradients propagated to inputs from different variables. As shown in Table 2, when the gradients of attention scores and standard deviation are retained in the sentiment classification task, the gradient norms reflected on the input are negligible. In the translation tasks, the weight is larger but still much smaller than that of group 2, which the decoder may introduce. Finally, it’s notable that the connection between our method and these experiments lies in the fact that the gradient produced by group 2 is equal to the transition matrix of each input in the decomposition D̄ (i.e., WX in Eq. (4)). Therefore, our decomposition indeed captures the major causalities of the model.
Overall, the results show that our approach is applicable and efficient in classification and endto-end generation. We provide additional results of AOPCs by different k in Appendix G, including an extra natural language understanding task from GLUE (Wang et al., 2018).

heading: 3.3 Ablation Study
text: We investigate the impact of different λ on the SST-2 and WMT14 datasets, which controls the interpolation of D̂ (signed) and D̂ (abs).
We achieve the best AOPC score with λ near 0.1 (Figure 2). Compared with the absolute-valuebased decomposition (λ = 1). The AOPC scores of
the pure signed-value-based decomposition (λ = 0) differ slightly from the best results. As the λ increases, the AOPC scores on both datasets decrease, demonstrating that improved consistency leads to better interpretability.

heading: 4 Applications
text: Our method can be applied to various scenarios by designing different partitions. In this section, we analyze the causes of model errors in sentiment classification and translation at the instance level. We set up our algorithm with D̂ (signed) for strict consistency. We also compare the results of IG (Integrated Gradient) and LOO (Leave-One-Out).

heading: 4.1 Errors in Sentiment Classification
text: We find that over half of the errors of the SST-2 test occur when the sentiment expressed at the sentence level is opposite to the polarity of the sentiment words in the input. For example, the sentence “if steven soderbergh’s ‘solaris’ is a failure it is a glorious failure.” is a positive comment, but the model’s prediction is negative.
Figure 3 shows the contribution heatmap generated by our algorithm and the baseline algorithms, where tokens belonging to the same word are divided into the same group for word-level explanations3. The results of the analysis show that the model focuses on both “failure” and fails classification, indicating the model’s insufficient understanding of the overall sentence meaning. It is notable that our method not only considers the last “failure” as the main basis of the model decision but the first “failure” as well. This is more intuitive since the model’s prediction only inverts as soon as both “failure” are masked. For comparison, the
3For other baseline algorithms, we sum the token-level scores within the group to obtain the group-level scores, despite of inconsistency.
other two baselines fail to indicate the impact of the first “failure”.

heading: 4.2 Errors in Translation
text: We noticed that, despite fluency in the target language, machine translation produces hallucinated outputs (Müller et al., 2020) that are semantically decoupled from the source sequence (Table 3).
We divide inputs into two groups to inspect their contributions to outputs: the source and the target prefix. Figure 4 shows the percentage of the contribution by target prefix at each generation step for the case in Table 3. Our algorithm indicates that the model tries to generate a sentence without accessing source information during hallucination since the target prefix dominates the contribution. On the
contrary, the contribution of the target prefix stays relatively low in a well-generated sequence. It only escalates at the generation of subword tails (step 7) or ⟨EOS⟩ tokens (step 9), where more language modeling takes over.
As a comparison, we did not find the above pattern in the results of IG. The results of LOO overestimate the contribution of the target prefix and lack interpretability of the trends on the well-generated sample. We further verify this pattern on more test samples, as shown in Figure 5. The contributions of target prefix to hallucinated samples are generally more than that to well-generated samples amongst all three methods, but only our algorithm distinguishes the two clusters.

heading: 5 Related Work
text: Interpreting DNNs involves various techniques, such as feature visualization (Olah et al., 2017; Yosinski et al., 2015), probing (Conneau et al., 2018; Shi et al., 2016), and analyzing learned weights (Tsang et al., 2018). Local interpretation belongs to another paradigm, which tries to interpret individual predictions of a DNN.
Existing works of local interpretation focus on assigning importance to individual features with respect to the prediction, such as pixels in an image or words in a sentence. The assignment employs methods like input occlusion (Li et al., 2016; Ribeiro et al., 2016b), gradientbased algorithms (Hechtlinger, 2016; Sundarara-
jan et al., 2017), layer-wise relevance propagation (LRP, Voita et al., 2021; Bach et al., 2015), decomposition-based methods (Murdoch et al., 2018; Singh et al., 2019; Jin et al., 2020; Kobayashi et al., 2021; Modarressi et al., 2022; Ferrando et al., 2022), and others (Hao et al., 2021; Shrikumar et al., 2017).
Specifically in NLP, Voita et al. (2021) extend LRP to the Transformer to analyze NMT models. Murdoch et al. (2018) introduces a contextual decomposition to track the word-level importance in LSTM (Hochreiter and Schmidhuber, 1997). Singh et al. (2019) extend the aforementioned to produce hierarchical clustering of words along with the contribution of each cluster.
Backpropagation-based algorithms such as gradient-based algorithms (Sundararajan et al., 2017) and LRP (Voita et al., 2021) have exponential time or space complexity, making their application on long sequences infeasible. The occlusion algorithms (Li et al., 2016; Chen et al., 2020) also suffer from performance degradation on long sentences since occlusion has a limited impact on the semantics of long sentences. Our methods are similar to those based on additive decomposition (Kobayashi et al., 2021; Modarressi et al., 2022; Ferrando et al., 2022; Mickus et al., 2022). Despite not being explicitly noted, these methods all rely on the same assumption to linearize attention scores and layer normalization. However, they do not decompose the FFN layer and instead use heuristic algorithms to aggregate contributions across layers.

heading: 6 Conclusion
text: In this paper, we find that specific DNNs satisfy linearity under proper assumptions. We further leverage the linearity of the model to generate local explanations. We test proposed algorithms with the standard and pretrained Transformer architecture on two benchmark datasets. Experimental results show that our method achieves competitive performance in efficiency and fidelity of explanation. Additionally, we offer examples of different tasks to apply our algorithms for error analysis. We leave the analysis of other DNNs and the intermediate states of the models as future work.

heading: 7 Limitations
text: Although based on the Transformer model, our methods also apply to various DNN modules, including CNNs, Poolings, and their compositions.
The applications of the proposed method in computer vision are left for future work.
An obvious limitation of this work is that we only verify our algorithm on models activated by ReLU. This issue can be alleviated because our algorithm is theoretically compatible with any piecewise linear activation function. For other functions in the ReLU family, such as the GELU (Hendrycks and Gimpel, 2016) used by BERT (Devlin et al., 2019; Liu et al., 2019), we replace the activations with ReLU, then fine-tune on downstream tasks and pretrain tasks (Appendix E). Our algorithms bog down on more complex nonlinear functions (e.g., sigmoid and tanh). It’s intuitive to fit these nonlinear functions with ReLU-activated FNNs. However, this leads to additional computational and space complexity, which degrades performance after fitting.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for their insightful comments and suggestions that helped us to improve the quality of this manuscript. Their feedback was invaluable in helping us to refine our ideas and present them more effectively. Shujian Huang is the corresponding author. This work is supported by National Science Foundation of China (No. 62176115, 62176120), the Liaoning Provincial Research Foundation for Basic Research (No. 2022-KF-26-02).

heading: A Consistency Condition
text: Proof. Prove the sufficiency and necessity of Lemma 1, respectively.
Sufficiency. To prove the sufficiency, we introduce decomposition under the elementary partition as an intermediate, where the elementary partition Pe is a partition in which each element in X forms a set., i.e. Pe = {{x1}, · · · , {xm}}.
For any two partitions Pa and Pb that Pa ∩Pb ̸= ∅ and g ∈ Pa ∩ Pb, if D satisfies Group Additivity, then there is
Dh
Dg ∣∣∣∣ Pa = ∑ x∈g Dh D{x} ∣∣∣∣ Pe
= Dh
Dg ∣∣∣∣ Pb .
(18)
Necessity. For any two groups g1, g2 ∈ X that g1 ∩ g2 = ∅ and g1, g2 ̸= ∅, and any partitions Pa and Pb that g1, g2 ∈ Pa, g1 ∪ g2 ∈ Pb. Without loss of generality, assume that Pa = {g1, g2, ga3 , · · · , gam} and Pb = {g1 ∪ g2, g b 3, · · · , gbn}.
There are two different partitions P ′a = {g1, g2, X\(g1∪g2)} and P ′b = {g1∪g2, X\(g1∪ g2)}. And we have
m∑
i=3
Dh
Dgai ∣∣∣∣ Pa = h− Dh Dg1 ∣∣∣∣ Pa − Dh Dg2 ∣∣∣∣ Pa , (19)
Dh
D(X\(gi ∪ gj)) ∣∣∣∣ P ′a = h− Dh Dg1 ∣∣∣∣ P ′a − Dh Dg2 ∣∣∣∣ P ′a .
(20)
By the consistency of D , we have DhDg1 ∣∣∣ Pa =
Dh Dg1 ∣∣∣ P ′a and DhDg2 ∣∣∣ Pa = DhDg2 ∣∣∣ P ′a . Thus
m∑
i=3
Dh
Dgai ∣∣∣∣ Pa = Dh D [X\(gi ∪ gj)] ∣∣∣∣ P ′a . (21)
Similarly, there is n∑
i=3
Dh
Dgbi ∣∣∣∣ Pb = Dh D [X\(gi ∪ gj)] ∣∣∣∣ P ′b . (22)
Now we get
Dh
Dg1 ∣∣∣∣ Pa + Dh Dg2 ∣∣∣∣ Pa = h− m∑
i=3
Dh
Dgai ∣∣∣∣ Pa
= h− Dh D [X\(gi ∪ gj)] ∣∣∣∣ P ′a .
(23)
Dh
Dgi ∪ gj ∣∣∣∣ Pb = h− n∑ i=3 Dh Dgbi ∣∣∣∣ Pb
= h− Dh D [X\(gi ∪ gj)] ∣∣∣∣ P ′b . (24)
Again according to the consistency, we have
Dh
D [X\(gi ∪ gj)] ∣∣∣∣ P ′a = Dh D [X\(gi ∪ gj)] ∣∣∣∣ P ′b .
(25) So
Dh
Dg1 ∣∣∣∣ Pa + Dh Dg2 ∣∣∣∣ Pa = Dh Dgi ∪ gj ∣∣∣∣ Pb . (26)

heading: B The Uniqueness of Interpretable Decomposition
text: We claim that the interpretable decomposition of linearly decomposable h is unique.
Proof. Assuming h = f(X) = ∑m
i W X i xi.
Based on Orthogonality, we have
Dxi Dg =xi for xi ∈ g, (27)
Dxj Dg =0 for xj /∈ g. (28)
By the linear transformation of Linearity, we have
D(WXi xi)
Dg = WXi Dxi Dg
= WXi xi for xi ∈ g, (29)
D(WXj xj)
Dg = WXi Dxj Dg = 0 for xj /∈ g. (30)
By the addition of Linearity, we have
Dh Dg =
D( ∑m
i W X i xi) Dg
= m∑
i
D(WXi xi)
Dg
= ∑
xi∈g WXi xi.
(31)

heading: C Mathematical Properties of D̂
text: By definition, it is clear that D̂ satisfies Linearity.
Proof. proof of Group Additivity by mathematical induction.
Base Case. The same as Eq. (14), D̂ degenerates to D̄ and therefore inherits the Group Additivity property.
Induction step. For any hidden state hl, it is obtained either by linear transformation and addition or by ReLU. Assume that the hidden states involved in the operation to get hl all satisfy Group Additivity.
For addition and linear transformation, without loss of generality, suppose h′ = W1h1 + W2h2, then there is
D̂h′ D̂g1 + D̂h′ D̂g2 = D̂(W1h1 +W2h2) D̂g1 + D̂(W1h1 +W2h2) D̂g2
= W1 D̂h1
D̂g1 +W2
D̂h2 D̂g1 +W1 D̂h1 D̂g2 +W2 D̂h2 D̂g2
= W1 D̂h1
D̂g1 ∪ g2 +W2
D̂h2
D̂g1 ∪ g2
= D̂h′
D̂g1 ∪ g2 .
(32)
For ReLU, suppose h′ = relu(h) = W ′h and Θ is the separated contribution in Eq. (11), i.e.
Θ := W ′ D̂h D̂B − relu( D̂h D̂B ). (33)
Then we have
D̂h′ D̂g1 + D̂h′ D̂g2 = W ′ D̂h D̂g1 + αg1Θ +W ′ D̂h D̂g2 + αg2Θ
= W ′ D̂h
D̂g1 ∪ g2 + (αg1 + αg2)Θ
= W ′ D̂h
D̂g1 ∪ g2 + (
D̂h D̂g1∑ g∈P D̂h
D̂g
+
D̂h D̂g2∑ g∈P D̂h
D̂g
)Θ
= W ′ D̂h
D̂g1 ∪ g2 +
D̂h D̂g1 + D̂h
D̂g2∑ g∈P D̂h
D̂g
Θ
= W ′ D̂h
D̂g1 ∪ g2 +
D̂h
D̂g1∪g2∑ g∈P D̂h
D̂g
Θ
= W ′ D̂h
D̂g1 ∪ g2 + αg1∪g2Θ
= D̂h′
D̂g1 ∪ g2 .
(34)
Notice that we apply the signed-value-based decomposition (Eq. (13)) in line 3 of Eq. (34), while the absolute-value-based one does not make the derivation to hold.

heading: D The Uniqueness of α
text: We claim that the signed-value-based α calculation is the only continuous solution that makes the decomposition D̂ satisfies consistency.
Proof. Since consistency and Group Additivity are equivalent, we will use both of their properties in the proof.
First prove that α itself satisfies Group Additivity, i.e, αg1 + αg2 = αg1∪g2 .
According to the Group Additivity property of D̂ , we have
D̂relu(h)
D̂g1 +
D̂relu(h)
D̂g2 =
D̂relu(h) D̂g1 ∪ g2 , (35)
W ′ D̂h
D̂g1 + αg1Θ +W
′ D̂h
D̂g2 + αg2Θ =
W ′ D̂h
D̂g1 ∪ g2 + αg1∪g2Θ,
(36)
W ′ D̂h
D̂g1 ∪ g2 + αg1Θ + αg2Θ =
W ′ D̂h
D̂g1 ∪ g2 + αg1∪g2Θ,
(37)
αg1 + αg2 = αg1∪g2 , (38)
where Θ is defined in Eq. (33). Suppose that α is calculated by the function A,
that is
αg = A(H, D̂h
D̂g ), (39)
where H = { D̂h D̂g |g ∈ P}. Next, we prove that the value of A(H, D̂h
D̂g ) is
only related to D̂h D̂g
and ∑
g∈P D̂h D̂g , instead of a specific values of other elements in H .
Since the sum of α is 1, we have
A(H, D̂h
D̂g ) = 1−
∑
e∈H,e ̸=g A(H,
D̂h D̂e ). (40)
By the Group Additivity of α,
∑
e∈H,e̸=g A(H,
D̂h D̂e ) = A(H ′,
D̂h
D̂ ⋃ e∈H,e ̸=g e ),
(41)
where H ′ = { D̂h D̂g , D̂h D̂ ⋃ e∈H,e̸=g e }. By the Group Additivity of D̂ , there is
D̂h
D̂ ⋃ e∈H,e̸=g e =
∑
g∈P
D̂h D̂g − D̂h D̂g . (42)
With Eq. (40) and Eq. (41), we have
A(H, D̂h D̂g ) = 1−A(H ′,
∑
g∈P
D̂h D̂g − D̂h D̂g ), (43)
and H ′ = { D̂h D̂g
, ∑
g∈P D̂h D̂g − D̂h D̂g }. The proposition is proved. Let’s replace the func-
tion A(H, D̂h D̂g
) with function A′( ∑
g∈P D̂h D̂g , D̂h D̂g ). Notice that we have
A′(s, D̂h D̂g1 ) +A′(s, D̂h D̂g2 ) =A′(s,
D̂h
D̂g1 ∪ g2 )
=A′(s, D̂h
D̂g1 +
D̂h D̂g2 ).
(44)
This means that A′(s, x1)+A′(s, x2) = A′(s, x1+ x2) always holds. Thus A′(s, ax) = aA′(s, x) holds for all a ∈ Z and all x, s ∈ R. Further, A′(s, abx) = a bA
′(s, x) holds for all ab ∈ Q and all x, s ∈ R.
Finally, we prove that A′(s, rx) = rA′(s, x) holds for all r ∈ R and all x, s ∈ R.
If r ∈ R and r /∈ Q, consider a sequence qi in Q converging to r. Then the sequence qix converges to rx and the sequence qiA′(s, x) converges to rA′(s, x). If A′ is continuous, then
A′(s, rx) =A′(s, lim i→∞ qix)
= lim i→∞
A′(s, qix)
= lim i→∞
qiA ′(s, x)
=rA′(s, x).
(45)
Therefore, A′(s, x) is a linear function with respect to x. Suppose A′(s, x) = cx, we have
1 = ∑
g∈P A′(
∑
g∈P
D̂h D̂g , D̂h D̂g ) =
∑ g∈P c D̂h D̂g = c ∑ g∈P D̂h D̂g ,
(46)
c = 1/ ∑
g∈P
D̂h D̂g . (47)

heading: E Experiment Details
text: Data preprocessing All input text of GLEU and IMDB datasets are encoded by Byte-Pair Encoding (BPE, Sennrich et al., 2016) of RoBERTa, containing 50K subword units of byte-level vocabulary.
For WMT14 En-De dataset, sentences have been jointly tokenized and byte-pair encoded with 32k merge operations using a shared vocabulary.
Training details For GLUE (Wang et al., 2018), we follow the hyperparameter settings of RoBERTa (Liu et al., 2019), with batch sizes ∈ {16, 32}, and learning rates ∈ {1e− 5, 2e− 5, 3e− 5}, with a linear warmup for the first 6% of steps followed by a linear decay to 0. We use the Adam optimizer (Kingma and Ba, 2015) with β1 = 0.9, β2 = 0.98 and ϵ = 1e − 6. We fine-tune 10 epochs in each dataset. More details about hyperparameter configurations can be found in https://github. com/facebookresearch/fairseq/tree/main/ examples/roberta/config/finetuning. For the IMDB dataset we set batch = 16, lr = 1e− 5 and warmup = 1256, other settings are the same as GLEU benchmark.
Since the GELU activation (Hendrycks and Gimpel, 2016) in RoBERta is incompatible with our theory, we replace it with ReLU at fine-tuning, which leads to performance degradation, especially with small datasets. This issue can be solved by fine-tuning pre-training tasks prior to the downstream tasks: we re-train the pretraining tasks (i.e., masked language modeling) on a smaller dataset with ReLU activation function. We adopt the WikiText-103 dataset as the retraining corpus and use the same training configuration as finetuning, including batch = 16, lr = 1e − 5 and warmup = 1500. The model with additional finetuning by pretraining tasks is comparable, and sometimes better than RoBERTa (Table 4).
For machine translation, we adopt β = [0.9, 0.98] and ϵ = 1e−8 for Adam optimizer. The learning rate linearly increases from 1e−7 to 7e−4 with 4000 warmup steps, then decay by the inverse square root schedule. We additionally adopt label smoothing at 0.1. Training instances are batched together by approximate sequence length. Input tokens in the batch are restricted to 8102 per GPU. The model is updated for 300k steps. We average the last 5 checkpoints, each of which is saved at the end of an epoch.
All experiments were trained and evaluated using a single RTX 3090 Ti GPU, except for the translation model, which was trained on 2 RTX 3090 Ti GPUs.

heading: F Performance Experiments
text: We present the full RoBERTa results of our implementation on development sets in Table 4. For IMDB, the fine-tuned RoBERTa model achieves 93.8% accuracy on the full test set. The model achieves a BLEU score (Papineni et al., 2002) of 27.19 on the WMT14 when trained from scratch.

heading: G Results of AOPCs changing with
text: different k
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
7
7 A2. Did you discuss any potential risks of your work? Our work contains little potential risk.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 2, 3, 4
3 B1. Did you cite the creators of artifacts you used? 3
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? All artifacts are publicly available and used in academic research.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We use it for research purposes only.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We have used only publicly available datasets whose sensitive information has passed the provider’s checks.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Documentation of our algorithms will be provided in the future along with the code.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 3
C 3 Did you run computational experiments? 3, 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix E
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 3, Appendix E
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? The experimental results are not randomized.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix E
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: ? Documentation of our algorithms will be provided in the future along with the code. 3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 3
C 3 Did you run computational experiments? 3, 4