Attention Flows for General Transformers
Authors: Niklas Metzger
Christopher Hahn
Julian Siber
Frederik Schmitt
Bernd Finkbeiner
Venue: Submitted to ICLR 2023 (2023)
Keywords: transformer
explanations
attention flow
shapley value

Abstract: In this paper, we study the computation of how much an input token in a Transformer model influences its prediction. We formalize a method to construct a flow network out of the attention values of encoder-only Transformer models and extend it to general Transformer architectures, including an auto-regressive decoder. We show that running a maxflow algorithm on the flow network construction yields Shapley values, which determine a player's impact in cooperative game theory. By interpreting the input tokens in the flow network as players, we can compute their influence on the total attention flow leading to the decoder's decision. Additionally, we provide a library that computes and visualizes the attention flow of arbitrary Transformer models. We show the usefulness of our implementation on various models trained on natural language processing and reasoning tasks.

Under review as a conference paper at ICLR 2023
ATTENTION FLOWS FOR GENERAL TRANSFORMERS
Anonymous authors
Paper under double-blind review
ABSTRACT
In this paper, we study the computation of how much an input token in a Trans- former model influences its prediction. We formalize method construct flow network out of the attention values of encoder-only Transformer models and extend it to general Transformer architectures, including an auto-regressive decoder. We show that running a maxflow algorithm on the flow network construction yields Shapley values, which determine player's 's impact in cooperative game theory. By interpreting the input tokens in the flow network as players, we can compute their influence on the total attention flow leading to the decoder's s decision. Additionally, we provide a library that computes and visualizes the attention flow of arbitrary Transformer models. We show the usefulness of our implementation on various models trained natural language processing and reasoning tasks.
INTRODUCTION
The Transformer (Vaswani et al., 2017) is the dominant machine learning architecture in recent years, finding application in NLP (e.g., BERT (Devlin al. 2019). GPT-3 (Brown et al. 2020), or LaMDA (Collins and Ghahramani (02T)) ) computer vision (see Khan et al. (2021) for a survey),
mathematical reasoning (Lample and Charton 2019 Han al. 2021). or even code and hardware synthesis (Chen et al. 2021; Schmitt et al. 2021). The Transformer relies on attention (Bahdanau
ct al. 2015) that mimics cognitive attention, which sets the focus of computation on a few concepts at time. In this paper, we rigorously formalize constructing flow network out of attention values (Abnar models, including a decoder. While theoretically yielding a Shapley value (Shapley 1953) quite trivially, we show that this results in meaningful explanations for input tokens' influence on the total flow affecting Transformer's prediction.
and Zuidema 2020) and generalize
Its applicability in various domains has made the Transformer architecture incredibly popular. Models are easily accessible for developers around the world, for example at bugginggface co (Wolf let al. 2019). However, blindly using fine-tuning these models might lead to mispredictions and unwanted biases, which will have a considerable negative effect on their application domains. The sheer size of the Transformer models makes it impossible to analyze the networks by hand. Explainability and visualization methods, e.g., Vig (2019), aid the machine learning practitioner and researcher in finding the cause of a misprediction revealing unwanted biases. The training method or the dataset can then be adjusted accordingly.
Abnar and Zuidema 2020) introduced Attention Flow as a post-processing interpretability technique that treats the self-attention weight matrices of a Transformer encoder as flow network. This technique allows analyzing the flow of attention through the Transformer encoder: Computing the maxflow for an input token determines the impact of this token on the total attention flow. Ethayarajh and Jurafsky (202T) discussed a possible relation of the maxflow computation through the encoder flow network to Shapley values, which is concept determining a player's impact in cooperative game theory and can be applied to measure the importance of a model's input features. However, the lack of clear formalization of the underlying flow network has made it difficult to assess the validity of their claims, which we aim to address this work.
We extend our formalization of the approach to a Transformer-modellaggee general decoder Transformers and decoder-only Transformers such as GPT models Radford et al. (2018). While, after applying a positional encoding, the encoder processes the input tokens as a whole, the decoder layers operate auto regressseess i.c., a sequence of tokens will be predicted step-by-step, and already predicted input tokens will be given input the decoder. This results in
technique, including

---

Under review as a conference paper at ICLR 2023
(a) With |||+1 node columns, |L] edge columns, I' {is) and = {1.3, 5, 6}. The red node depicts the input token for which the maximum flow is computed. The blue node represents the terminal node t.
(b) Input token set O' fou} and embedding t, where the output token os is currently predicted. The first "input" token, i.e. 0r is the special start token of the decoder.
Figure I: Encoder attention in a flow network (left) and decoder attention in a flow network (right).
significantly different shape of the flow network and, in particular, requires normalization account for the bias towards tokens that were predicted later than others. We account for the auto-regressive nature of the decoder by ensuring positional independence of the computed maxflow values. We implemented our constructions In summary, our contributions are the following. We formalize encoder-only attention flow and generalize the approach to encoder-decoder and decoder-only Transformers in Section 2 Furthermore, we use the formalization to construct an explicit algorithm for attention flow computation and analyze its complexity. In Section we show that the computed attention flow values are Shapley values for all three architectures. Section Mirioon/jitrrduue introduussss a tool to compute and visualize attention flow for arbitrary Transformers. We report on qualitative and quantitative experiments that show the effectiveness of our approach, including token bias and single-head attention analyses.
Python library, which we will publish under the MIT license.
Related Work. We would like emphasize the work which we build: Abnar and Zuidema (2020)
introducing attention flow for Transformer encoders and [Ethayarajh and Jurafsky| (2021) drawing possible connection between encoder attention flows and Shapley values. An explainability overview is given by Samek et al. (2017) and Burkart and Huber (2021). An overview over Shapley value
formulations for machine learning models is given by SSundararaaan and Najmil (2020), which are not
restricted to Transformer models and do not include attention flow (Lindeman 1980 Grömping 2007 Strumbelj et al.,
Owen 2014; Owen and Prieur 2017
2009
Strumbelj and Kononenko 2014 Datta
al. 2016 Lundberg and Lee
2017
Lundberg al.
2018 Aas al.] 2019 Sun and Sundararajan
2011 Sundararajan et al. 2017 Agarwal et al. 2219))
Shapley values are also used for the val valuation
of machine learning data (Ghorbani and Zou, 2019).
Raw attention values can be visualized, e.g., Vig
(2019) and Wang et al. (2021). Chefer et al.
2021
assign local relevance based on the Deep Taylor
Decomposition principle (Montavon et al. 2017).
2 ATTENTION FLOW
Attention Flow (Abnar and Zuidema 2020) is post-processing interpretability technique that treats the self-attention weight matrices of the Transformer encoder as a flow network and returns the maximum flow through each input token. Formally, flow network is defined as follows.
Definition 1 (Flow Network). Given a graph G = (V, E), where V is a set of vertices and E S VXV set of edges, a flow network is a tuple (G, c, (), where e E - Roo is the capacity function and s and I. are the source and terminal (sink) nodes respectively. A flow is a function f F - TR satisfying the following two conditions: Flow conservation: Ye E VV, 1). 22((1) 0, where f V + IR is defined as x (u) = Evev f(v u), and capacity constraint: Ve E E. f(e) < c(e).
The value of flow [f] is the amount of flow from the source node to the terminal node t: [f| Evs(,.)EE feu- For a given set K of nodes, we define [ as the flow value from s to t only passing through nodes in K: If(K) Ev.(...j fxv. We define (t)) to be the total outflow value of node and (+) to be the total inflow value of node In optimization theory, the maximum flow problem mar() (1 (f)) (Harris and Ross 1955) is to find flows that push the maximum
possible flow value |s| from the source node to the terminal node which we denote by f fmaz-
2

---

Under review as a conference paper at ICLR 2023
2.1 ENCODER ATTENTION FLOW
Given encoder-only Transformer model, such the BERT (Devlin et al. 2019) model family, with H attention heads, L layers, M input tokens 1 = .i......... iM and the resulting self-attention ten- sor A E E IR x2 M For some X E N, we define [X] as the set (1)....X). For a set of positions J, a subset of input tokens I' C I and subset of heads H' C H, we construct a flow network Feno (A AI, ,I' = (G,c, e, 8, t) as follows:
1 E Alins
d' (iei+))1))
= (1 1+ (s,
d(i,,),)) = H 11=
d'=t
=: (((i)))) (ix,1 1)) i EIAE (+0]))(((0,,,, ((,,((())) e)
We visualize this flow network translation in Figure la The flow network consists of L. + columns of nodes and L columns of edges. The attention values are encoded as capacities on the edges. Thus the underlying graph of the flow network requires one additional column of nodes. Computing the maximum flow through this network determines the contribution of the input tokens I' to the attention flow towards the final encoder embeddings given by JJ. Note that the nodes in columns greater than 1 correspond to encoder embeddings and can not be interpreted as input tokens anymore. Residual connections can be taken into account as proposed by Abbar and Zuidema (020). i.c., by adding an identity matrix 7 and re-normalize it as 0.5A 0.57. By setting the start node successively to singleton sets containing only a single input token and all final embeddings to t, we can compute the encoder flow for every encoder input token as introduced by Abnar and Zuidemal (2020). The encoder flow network construction can also be used for models including classification task (see Section 4). To determine the influence of input tokens on the attention flow towards deciding the class, the terminal node t is only connected to the final embedding of the classification token.
2.2 DECODER ATTENTION FLOW
Generative Transformer models that involve decoder require a significantly different shape of flow network. We begin by investigating decoder-only models, with H attention heads, L layers, N "out- put" tokens O fo....... .N) and the self-attention tensor A <: = Since we consider decoder-only models, a prefix subset Oinput S O will be given as a problem input to the neural network model. Note that the first output token is always a special start token. For set of output tokens O' C, the position n of output token On E O and subset of heads H' C H, the construction of a flow network Fdco (A O', n) = (G,e, 8, () follows the structure of the decoder self-attention:
V = 0= 0 [L+1]
E: ((o,,0) (ou, 1 + 1)) o5:0N EOAEE []
H'
c((oj, 1), (ox, 1 1))
AJJJ
8 s: (== (d) d'=
t: (On L L+1)
H
We visualize the construction in Figure 1b Because of the auto-regressive nature of the Transformer decoder, we compute the maxflow to the last embedding of the decoder as this embedding will be used in the Transformer normalization to account for the bias towards tokens that were predicted later than others (later predicted tokens have more incoming edges). Intuitively, we require that the maxflow computation for any sub flow network F' constructed from the decoder flow network F to be independent of the absolute position of F' in F. Formally, assuming A 1 to have the same value e for every entry, i.c., the capacity of every edge the resulting-flon flow network is fixed to c, we require for every position n that Vom E O. maxflow (Faee (A {. n)) = c, which we call positional independence. We ensure this by dividing the result of max flow computation for given start token Om and end token on by 1+ (0 - (n - m)) - m. For a subset O' C O and a position n (where Vo E O'.m < n) and heads H', we can thus compute the influence of the token set O' to the total attention flow towards the embedding that predicts the n-th token, no matter whether it served as part of the problem input or is already predicted output token.
predict the next token. The auto-regression, however, requires
3

---

Under review as a conference paper at ICLR 2023
2.3 ENCODER- -ECDDER ATTENTION FLOW
For Transformer models consisting of an encoder and decoder, we combine both flow network translations with the encoder-decoder attention. Sioure the structure of the flow network for Transformer model with an encoder (top) and decoder (bottom). The last nodes of the flow network corresponding the final embedding of the encoder are, following the Transformer architecture. connected to every node layer of the network corresponding to the decoder. We omit some encoder- decoder edges for better visualization. Given a Transformer with attention heads, I. layers, M input tokens I = (io.....dxy (M). }. N output tokens O = {ou.- ON ), and resulting encoder self-attention tensor A R MIXMMMMM decoder self attentttnnion tensor A LxXNNN and encoder attention tensor A E R "I and subset of heads H' C H. we construct a flow network F(A R A A 1 n) = (G, (G,,,,,1 from flow networks Fene 11, = ((Vene, Eenc), Genc: Sence tenc) and Faec (A" 99,m) = Fdec(A
For a set of input tokens I', the position n of output token On
((Vdee) Edec), Cdecs ddees tdec) follows:
= Venc U Vdec Us 1: = (On: L + 1)
E Fenc Pates U [((i}, + 1); in ETAVE Vdee}
U (((emer (0m, 0)) Om EIAm <n}
Cene(U, D')
v (ig, 1), (ik:
i E 1
v (o 1)) v' = (ox, l'),
du,
Oj. EO
1 N7 E
AC
v = (Lz, 1+1),
h.l.j
y' 1), E 1, E
s, v' El
where le denotes a layer from the encoder and Uy denotes a node from the decoder. Again, we have to normalize to account for the auto-regressive bias, i.... require positional independence. For a given set on input tokens I' and heads H²' we can thus asses the contribution of this set to the total attention flow towards the emhedding that predicts the n-th token by computing the maxflow through this network. If one is interested in the influence of an already computed output token Om. where m < n, on the prediction of on, then the construction for the decoder-only case in Section 2.2 applies.
2.4 ALGORITHM
The flow network constructions can be directly used in an algorithm to compute the attention flow for input to- kens. Algorithm shows the algorithm for computing the attention flow for every input-output token pair. We build the flow network for every pair and compute the maxi- mum flow in the corresponding network with the Edmonds- Karp Edmonds and Karp (192)) algorithm. The runtime of Edmonds-Karp is in (VE2), where the edges and nodes V are given by the number of layers and onput-output tokens. Since we run this algorithm for every input-output pair (with only partially rebuilding the flow network), we additionally gain linear complexity in the number of input and output tokens. We evaluate the implementation of this algorithm and its variations for mnlder-ooly and decoder-only Transformers in Sec.
Input: A A", A I,, Output: f:0 x +R
f = None
for o E O do
for do f(o. + Ed. AP, A,AC, (i),o)
return Algorithm Attention flow.
2.5 OPTIMIZATIONS
The flow network constructions apply to subsets of heads, especially single heads. The results of head computations are joined using linear projection, so each head has access to the computations of all heads in the previous layers. The task of a head in layer can be independent of its task in previous layers l' < I. In practice, however, heads are biased towards keeping their respective tasks, such that we also found good interpretability results by considering the attention flow of attention heads independently (see Section 4). A flow network can be constructed for a single head by following the above constructions, setting H' to every singleton. If the computation time of the maxflow for large
4

---

Under review as a conference paper at ICLR 2023
Figure 2: Sketch of the Encoder-decoder attention flow network for input token is and embedding t, which is used to predict Og. Encoder-decoder connections are sketched for the first node.
Transformer models exceeds time limits, relaxations on the flow network possible. First, note that the flow network only needs to be constructed once. As expected, the computation time of the maxflow in the network constructions increases with larger input and output sequences. Running time can be traded against heuristically shrinking the size of the flow network. This can be done in two dimensions. Following the practical assumption that heads often keep their tasks throughout subsequent layers, the first dimension is to shrink the flow network on the æ-axis. This can be done by simply skipping some of the inner layers of the network by merging layers by taking the average of the raw attention values across layers as capacities. Furthermore, the network can also be shrunk in the y-dimension similarly by grouping input and output tokens. For example, tokens predict and ed can be combined into one node.
3 SHAPLEY VALUE EXPLANATIONS
In this section, show how the extended flow network constructions the Transformer decoder FAee O', n) and F(A=, AD A I', n induce Shapley value explanations for the tokens of the input sequence. The Shapley value (Shapley 1953) solution concept determining the impact of player in cooperative game theory and an increasingly popular concept to determine the influence of certain input features on model's decision.
game with transferable utility (TU) a pair (P,u), with (1.....p} being finite set of players and v 2" + R. being the payoff function.
Definition
A subset S C P is called a coalition. The payoff function assigns every coalition of players S a real number (S) E with v(0) = The share of a player of the allocated payoff is (v)(). The encoding of the attention values represents a player and the total flow through the network represents the total payoff (Ethayarajh and Jurafsky 2021). The Shapley values of the players in TU game formally defined as follows.
flow network is TU game. A node in the flow network
Definition 3 (Shapley Value). Let II(P) be the set of all player permutations and let E II(P) be permutation of players. Let all players ahead of a player i be defined as Pe((T) = {i e P (() < (1)). The Shapley value lo defined as the share of payoff for a given player P: (P, = p! Ee (P( (i}) (()))
From a game-theoretic viewpoint, Shapley values are well-suited for determining the payoff share that players deserve, as the values satisfy the desirable properties efficiency, symmetry, null player, and additivity. The mathematical definition of the properties can be found App. These properties above also responsible for making Shapley values attractive approach for explaining model's decisions, i.e., features that do not contribute to the accuracy model should be null players, and features that contribute equally should satisfy symmetry.
(Decoder- Only Flow Is Shapley Value). Consider a Transformer decoder with Н attention heads, L layers, N "output" tokens O = {oi...., , and the self-attention tensor AD E B Let Jnan be the maxflow computed in the flow network Face (A {0), n) as defined the previous section. Consider the TU-game (P, v). where the players E P -1 [1..... N} correspond nodes (op, 0) from the first layer of the Transformer decoder. For given coalition S C
Propusition
P. let the value function be v(S) = E.es forar i.e., the sum of max-flows of nodes corresponding S. Then, the max-flow foia for some p is its Shapley value.
5

---

Under review as a conference paper at ICLR 2023
12000
00031
Ng
0L006
0:341
O 79449
0.8/07
Akk
64083
0.311
14
0/066
0,7716
0,7517
0,7650
0,6049
6.449
0.6248
0,5018
0:297
0.523
0,771
8:4034
E44042
(L3128
6688
8466
#ocstis Bay
Altstiinn fow exokder
Attinns Sow encodar
2,0000
2,0000
saunic fire
0,300
1,9070
2,0015
aetiio flow decoder
1terrtoo flow decoder
(a)
(b)
Figure 3: Heatmap of the attention flow of the GPT-2 model after 1, 4, and 6 predicted tokens in (a)
and Heatmap depicting the attention flow for unimportant token detection in SAT assignments in (b).
The proof immediately follows from the fact that every max-flow of some node S is an independent computation and the payoff of a coalition is defined as a sum of these independent contributions, which trivially qualifies value is trivial, we show in our experiments in the following section that the maxflow computation indeed yields meaningful explanations for the network's 's attention flow.
Shapley value. Although this theoretical correspondence to Shapley
Note that our line of reasoning significantly differs from Ethayyrajl and Jurafsky (202I). In particular, we compute a separate max-flow for every token in the set of players. This is because key assumptions about flow networks that they make in their proof do not hold: They argue that as long as nodes come from the same layer, blocking flow through some of these nodes does not change the possible flow through the others, such that they can deduce that the utility player adds when joining coalition independent of the identity of the players already in the coalition. However, this not the case: Several nodes from the same layer can compete for capacity downstream in the network even if they have no direct connection, e.g., if we have two tokens 0 O2 in one layer each attended with 0.5 attention by node 03 which itself is only attended to with 0.5 attention. Now, the utility or adds upon joining a coalition as defined by Ethayarajh and Jurafsky (202T) does depend on whether 02 is already part of it. We deduce from the above discussion that it may violate the symmetry of a Shapley value, as the payoff for 0j and 02 can be unequally allocated.
The ideas outlined for Proposition I also apply to the encoder-decoder attention flow. In the following, let finur be the maxflow computed in the flow network construction F(A A A {d). n) over the Transformer with H attention heads, L layers, M input tokens I = fio..... JM). N output tokens O- = (oo..... ON }. and resulting encoder self-attention tensor AE E x KM decoder self-attention tensor AD E Hx and encoder-decoder attention tensor A E &RHXLXNXM,
Flow Is a Shapley Value). Consider the TU-game (P, t). where the players p P = {1, (1.... } correspond nodes (ip. 0) from the first layer. Let the value function or a given coalition S C P be defined as v(S) Eees finm i.e., the sum of max-flows of nodes corresponding to S. Then, the max-flow finaz for some p E P is its Shapley Value.
Corollary EEncoder-Decodee
4 EXPERIMENTS
In this section, we report on natural language processing and logical reasoning experiments. We implemented the algorithm from Section _ The architectural details of the models are shown in Table lb We visualize the maxflow attention values in heatmaps, lineplots. and violinplots (see, for example, Figure 3b). The maxflow is computed with NETWORKX [Haberg et al. (2008), and the heatmaps comparing the attention flow from input/predicted token current predicted token visualized with SEABBRN Waskom (02TT) The heatmaps are either showing only the attention flow from input tokens if the model is encoder-only (enc.), separated into different heatmaps for input tokens and auto-regressive tokens for encoder and decoder (enc. + dec.), or show one heatmap for all tokens if the architecture is decoder only (dec.). Higher values represent higher attention flow.
'The code and experiments will be published after the double-blind review phase.

---

Under review as a conference paper at ICLR 2023
606
1.0
0.8
sum attertionn
800
0.8
04
801
000
(b)
Figure 4: The attention flow for every head of GPT-2 separately in (a) and the sum of all attention flow values per head for 300 sampled input queries on GPT-2 (b).
4.1 TOKEN RELEVANCY IN DECODER FLOW
Text completion. We demonstrate that our technique captures the attention flow changes during auto-regressive decoding. For this experiment, we track the attention flow changes in GPT-2|Radion ict al. (2019) while decoding the predicted tokens. The input sequence to this decoder-only mode was "My name is John, my profession is". Figure 3arteraaaaa depicts the attention flow after decoding the first, fourth, and sixth tokens. The resulting flow network can be found in Figure L3 in the appendix. Generally, GPT-2 models attend the first token the most (ef. Section 4.3) The differences in the attention flow are visible as the attention flow on previous tokens different for each decoding step. Most notably, the attention flows shift heavily toward the token "profession" when predicting the token "doctor". We observed these heavy switches in decoder attention flow values throughout our experiments, which is why this approach is a valuable addition existing analysis methods. The computation time of the flow values for this example only took 1.38, 1.50, and 2.09 seconds.
Satisfying assignments for SAT. In this experiment, we considered the problem of computing satisfying assignment to propositional logical formula. A formula in propositional logic is constructed out of variables and Boolean connectives - (not), (or), A (and), + (equivalence). For example, let the following propositional formula be given: & V (a A -a). A satisfying assignment is a mapping from variables to truth values, such that the formula evaluates to true. For example, satisfying assignment for the formula above is the mapping {b 1, + 0). The variable a, however, has no impact on the truth value of the formula. As long as is set to 1, can be predicted either or O. We conducted experiment to detect parts of the propositional formula that have no impact on predicted assignments. We trained a Transformer with an encoder and decoder to predict satisfying assignments. The attention flow values for the following two propositional formulas are depicted in Figure 3bb PropSAT, := bV (a A -a) in tokens: (a&Lla and PropSAT, == (a -a) b in tokens: (a&cla) (b. The disjunet (a A -a) plays role any satisfying assignment since any mapping of a results in this subformula being false. Regardless of the position in the formula, the flow computation of the network detects this as unimportant: the inputs to the encoder and -a have significantly less influence to the total attention flow than b.
(implication), and
4.2 HEAD TASK ANALYSIS
LTL trace prediction. We experimented with predicting satisfying traces linear-time temporal logic (LTL) (Pnucli 1977). We used Transformer trained on this task by Hahn et al. (2021). LTL generalizes propositional logic with temporal operators such as O (next) specify the behavior of systems that interact with their environments over time. An LTL formula is satisfied by a trace, which an infinite sequence of propositions that hold at discrete timesteps. We finitely represent satisfying traces to LTL formulas brackets. For example, the LTL formula O(a A O-a) denotes that in the second position, a must be true, and in the third position must be false. The model correctly predicts the trace, where the first position and the loop are arbitrary and hence set to true: trace 1; -a; [1]. Figure (5b depicts the maxflow computation for two heads. The left head focuses the O -a part of the formula and
(until) and is used to
prefix, followed by a loop, denoted by curly
7

---

Under review as a conference paper at ICLR 2023
0.2647
0.2857
0.2838
0.2568
0,2657
X
0,2855
0 2.200
0.2161
atterbu flow
04
0.2837
0.2147
encoder
attention
010902
0.1445
0.2
O. 1021
0.1431
1064
01042
0.1
0,0851
Input Soken
attertoo flow decoder
attention flow decoder
(a)
(b)
Figure 5: Violinplot for the distribution of attention flow of GPT-2 for 500 samples in (a) and Heatmaps for two heads of the LTLSat model, each attending a different timestep in (b).
the third position of the trace where is not allowed. The right head focuses on the left conjunct a, which must appear on the second position of the trace (see Appendix B for another example).
Translation. this experiment, we used the OPUS-MT-EN-DE model Tiedemann and Thottingal (2020) for translating between English and German. The input sentence is "The pilot lost her suitcase.", which is translated to "Der Pilot hat ihren Koffer verloren". The computed flow network can be found in Figure 12 While the meaning of the original sentence is ambiguous, as "the pilot" could be male or female, the translated sentence not, since the German phrase Der Pilor means a male pilot. It has been conjectured that such gender-bine biased translations can facilitate problematic stereotypes (Bolukbasi et al. 2016). Our analysis technique allows further insight into the internal of
mechanics of the Transformer model in such scenario. We analyze the task of the heads, them are shown in Figure 6 By computing the attention flow for the encoder and decoder, we can observe that the depicted heads solve opposing tasks: The head the left-hand side attends pilot lost her in the encoder and Der Pilot in the decoder, which is the one-to-one translation, but without corresponding possessive pronoun. The head on the right-hand side attends pilot and suitcase in the encoder and Pilot hat as well as Koffer in the decoder. Hence, from the attention flow, we can see that second head has little influence on the biased translation, as neither her, nor Der and ihren (the German pronoun corresponding to her) receive significant attention. This approach, therefore, gives us a helpful hint that we have to analyze the first head to get to the root of this biased translation.
Head attention. We analyze the influence of each head of GPT2 based on their contribution to the attention flow. Figure 4a shows the attention flow for each token and head for the input and output sentence "My name is John, my profession is be doctor: I am doctor of medicine." Heads 0, 1, and 2 show high and diverse attention flow values for different tokens, whereas all other heads have shallow and stable attention flow values. To explore this further, Figure shows the accumulated attention flows for all tokens and each head for 300 random samples. It supports the claim that the first three heads have higher attention flow values than all other heads.
Network
Heads
Layers
Architecture
DialogPT-medium (MIT)
16
24
dec.
OPUS-MT-EN-TT (MIT)
8
Input
Negative
Neutral
Positive
PropSat (MIT)
4.4
dec.
John is killer.
0.9548
0.0417
0.0034
LTLSat (MIT)
4
4,4
dec.
John good killer.
0.8949
0.0967
0.0084
GPT-2 (MIT)
12
12
dec.
John good killer
0.0981
0,3166
0,5853
RoBERTa (MIT)
12
12
enc.
(a)
(b)
Table 1: Results of the sentiment analysis in (a) and the parameter overview of the models in (b).
8

---

Under review as a conference paper at ICLR 2023
The
066
0.1250
0,1591
0.1250
pllor
pllc
lost
0. 1923
Pikt
0.1266
lost
Pilc
her
0.2052
bat
0.0799
her
0.1949
hat
0.1217
0.1 0.1756
iiren
n.0508
0.2149
ilecn
0,0290
IL0520
0.2149
Koffer
<55
0.21173
verfores
0,2149
sticttins cnoode
0.0082
enonda
0.009)
attetion fow decoder
anennicn flw decoder
Figure 6: Heatmap for two heads, divided in encoder and decoder. The left head attends the pilot, the head on the right the suitcase.
4.3 BIAS DETECTION
Sentiment detection. The flow analysis can be used to detect biases in the Transformer models. In this experiment, we used RoBERTa [Lu et al. (2019) finetuned for sentiment analysis on the TweetEval Barbieri et al. (2020) benchmark and computed the influence of input tokens the total flow deciding the classification. We computed the attention flow values for the input tokens of the following sentences and their results, shown in Tab. La The resulting flow network
be found in Figure T ji the appendix. While the first two sentences, "John a killer." and "John a good killer." correctly labeled with negative sentiment (even when having the adjective "good" in the sentence), having an emoji in the sentence immediately shifts the sentiment to be (falsely) labeled as positive. The computation of the attention flow is visualized in Figure 10 in the appendix. For the first two sentences, the attention on killer is the highest, considering only non-special tokens. Although the same holds for the third sentence, i.e., the attention flow denotes killer as the most important word, the fow-attended smiley changes the sentiment to positive. When computing the attention flow for each head, observe heads with attention flow 1.0 the emoji (see Figure in the appendix).
First token bias. While analyzing the attention flow of the decoder-only Transformer DialogPT Zhang al. (2020) and GPT-2 Radford al. (2019), we observed heavy bias toward the first decoded token (see Figure 3a and Figure 9 in the appendix). We computed the attention flow for 500 random samples of the OPUS MTTUS-MTEEN EN-DE test set. The results are visualized in Figure The first token contributes the most to the total attention flow regardless of the input tokens. Since the DialogPT model was trained on a dataset mined from reddit com, it might be beneficial to overattend the first token as many conversations on reddit com consist of concise sentences even single words. One should be aware of this bias when applying this model outside of similar domains.
5 LIMITATIONS AND CONCLUSION
The main limiting factor of this approach is that the attention flow in Transformer is the largest but not the only factor for deciding the next token prediction. Additionally to the many residual connections (which be incorporated into the flow networks; see Section 2). Transformer models contain feed-forward networks used as intermediate steps. Another minor caveat is that flow values cannot be compared across different model architectures as their absolute values have meaning. The values can solely be compared to other tokens in the same layer of the same model. This approach should thus be seen as a valuable addition (not replacement) the large toolbox for interpreting machine learning models. It generalizes the efforts in visualizing and interpreting raw attention values and attenion rollout. During our experiments, we found the attention flow values computed with the presented approach instrumental in analyzing models, finding biases, and fixing respective datasets.
To conclude, we formalized and extended the technique to construct a flow network from the attention values of encoder-only Transformer models to general Transformer models, including an auto- regressive decoder. Running a maxflow algorithm on these constructions returns Shapley values that determine the impact of token on the total attention flow leading the decoder's 's decision. We provide an implementation of our approach that can be applied to arbitrary Transformer models. Our experiments show this analysis method's 's applicability in various application domains. We hope implementation and constructions presented this paper will aid machine learning practitioners and researchers in designing reliable and interpretable Transformer models.

---

Under review as a conference paper at ICLR 2023
6 REPRODUCIBILITY STATEMENT
The supplementary material of this submission includes python notebooks to reproduce the figures presented in this paper with their underlying data. The code, datasets, models, and our notebooks for the reproduction of the experiments will be made publically available once the double-bling reviewing process ends.
REFERENCES
K. Aas, M. Jullum, and A. Løland. Explaining individual predictions when features are dependent: More accurate approximations to shapley values. arXiv preprint arXivv:903. 2019.
S. Abnar and W. H. Zuidema. Quantifying attention flow in transformers. In D. Jurafsky, Chai, N. Schluter, and J. R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 3190-4197 Association for Computational Linguistics, 2020. doi: 10.18653/v 1/2020.acl main.385. URL https://dol oxg/10.186533v11 acw.:2200.cc--o. maain.mainn..... 385in.385
A. Agarwal, K. Dhamdhere, and M. Sundararajan. A new interaction index inspired by the taylor series. 2019.
D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL [Rtttt///aaxxx org/ abs 1409. 0473
Barbieri, J. Hamacho-Collados L. E. Anke, and L. Neves. Tweeteval: Unified benchmark and comparative evaluation for tweet classification. In T. Cohn, Y. He, and Y. Liu, editors, Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL. pages 1644-1650. Association for URL https
Computational Linguistics, 2020. doi: 10.118633v11/0000 http://///22.....nnnnnoooo
7/doi. org/ 10. 11653/v1/202222 findings emnlp. 148
T. Bolukbasi, K.-W. Chang, J. Zou, V. Saligrama, and A. Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, page 4356 4364, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Bal- can, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Newral Information Processing Systems 2020, NewriPS 2020, December 6-12, 2020, virtual, 2020. URL http:tt://pooceeetttttt wew.riinnnn..uu.ssppeee.m neurips cc/ppapee/222///aalll Ttt.://......................... Abst html
N. Burkart and M. F. Huber. A survey the explainability of supervised machine learning. Journal of Artificial Intelligence Research, 70:245-317, 2021.
H. Chefer, S. Gur, and L.. Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 782-791, 2021.
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
E. Collins and Z. Ghahramani, 2021. URL http:///boog.cooee//cccocooo
lamda/
10

---

Under review as a conference paper at ICLR 2023
A. Datta, S. Sen, and Y. Zick. Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems. In 2016 IEEE symposium on security and privacy (SP), pages 598-617. IEEE, 2016.
Devlin, M. Chang, K. Lee, and K. Toutanova. BERT; pre-training of deep bidirectional transformers for language understanding, In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACI-HILT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 4171-4186 Association for Computational Linguisties, 2019. doi: 1/1999111199-4223 1423. URL
wtt.://.o0oo1111111////011111/ -1233
1423
Edmonds and R. M. Karp. Theoretical improvements in algorithmic efficiency for network flow problems, J. ACM, 19(2)244--66 264, 1972. doi: 10.1145/3216943321699 URL https
//doi org 10. 1145/321694 321699
K. Ethayarajh and D. Jurafsky. Attention flows are shapley value explanations. C. Zong. F. Xia, W. Li, and R. Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACI/LICNLP 2021, (Volume 2: Short Papers), Virtual Event, August 1-6, 2021, pages 49-54. Association for Computational Linguistics, 2021. doi: https://doi.crz 11t.0..16631111111 acl-short.........
htt.5//////222.allshhoooo URL
A. Ghorbani and Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning, pages 2242-2251. PMLR, 2019.
U. Grömping. Estimators of relative importance in linear regression based on variance decomposition. The American Statistician, (2): 132::1399447, 2007.
A. Hagberg, P. Swart, and D. S Chult. Exploring network structure, dynamics, and function using networkx. Technical report, Los Alamos National Lab. Los Alamos, NM (United States), 2008.
C. Hahn, F. Schmitt, J. U. Kreber, M. N. Rabe, and B. Finkbeiner. Teaching temporal logics neural networks. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL RRI.htps:: https:://openrere wwt. forum? 1d=d0cok- #4byz
J. M. Han, J. Rute, Y. Wu, E. W. Ayers, and S. Polu. Proof artifact co-training for theorem proving with language models. arXiv preprint arXiv:2102.0622 2021.
Harris and F. Ross. Fundamentals a method for evaluating rail net capacities. Technical report, RAND CORP SANTA MONICA CA, 1955.
S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah. Transformers in vision: survey. arXiv preprint arXiv:2 100.00 2021.
G. Lample and F. Charton. Deep learning for symbolic mathematies. arXiv preprint ar 2019.
R. H. Lindeman. Introduction to bivariate and multivariate analysis. Technical report, 1980.
Y. Liu, M. Ott, N. Goyal, J. Du M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692 2019. URLAttp://aazet
org/ abs /tlp///aa//11/// 110//997.1111 11692
S. M. Lundberg and S.-I. Lee. A unified approach interpreting model predictions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach. R. Fergus, S. Vishwanathan. and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 4765-4774, Curran Associates, Inc., 2017. URL
.earrrsniiiio nips. cerpsscc/pp.. htcppppprrr modee -predi pdf
S. M. Lundberg, G. G. Erion, and S.-L Lee. Consistent individualized feature attribution for tree ensembles. arXiv preprint arXiv: (000::802000500 038888888, 2018.

---

Under review as a conference paper at ICLR 2023
G. Montavon, S. Lapuschkin, A. Binder, W. Samek, and K.-R. Müller. Explaining nonlinear classification decisions with deep taylor decomposition. Pattern Recognition, 65:211-222, 2017.
A. B. Owen. Sobol'indices and shapley value. SIAM/ASA Journal on Uncertainty Quantification, (1):245-251, 2014.
A. B. Owen and C. Prieur. On shapley value for measuring importance of dependent inputs. SIAM/ASA Journal on Uncertainty Quantification, 5(1):986-1002, 2017.
A. Pnueli. The temporal logic of programs. In 18th Annual Symposium on Foundations of Computer Science, Providence, Rhode Island, USA, 31 October November 1977, pages 46-57. IEEE Computer Society, 1977. doi: 10.1109/SFCS.1977. /tL/Lttps///o...o///.0000
SFCS. 1977.37 32
A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. Improving language understanding by generative pre-training. 2018.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
W. Samek, T. Wiegand, and K.-R. Müller. Explainable artificial intelligence: Understanding, visual- izing and interpreting deep learning models. arXiv preprint
2017.
F. Schmitt, C. Hahn, M. N. Rabe, and B. Finkbeiner. Neural circuit synthesis from specification patterns, CoRR, 10ss/100..11664 2021. URL htttts:://arrttti www.aaiwww.....oeaaaaoooooaaoret http://wwrggaaab//a///ee abs/2a/7....../ 11864
L. Shapley. A value fo n-person games. Ann. Math. Study28, Contributions to the Theory of Games, ed. by HW Kuhn, and AW Tucker, pages 307-317, 1953.
E. Strumbelj and I. Kononenko. Explaining prediction models and individual predictions with feature contributions. Knowledge and information systems, 41(3):447-665, 2014.
E. Strumbelj, I. Kononenko, and M. R. Šikonja. Explaining instance classifications with interactions of subsets of feature values. Data & Knowledge Engineering, 68(10):886-90) 2009.
Y. Sun and M. Sundararajan, Axiomatic attribution for multilinear functions. In Proceedings of the 12th ACM conference on Electronic commerce, pages 177-178, 2011.
M. Sundararajan and A. Najmi. The many shapley values for model explanation. In International Conference on Machine Learning, pages 2269-9278. PMLR, 2020.
M. Sundararajan, A. Taly, and Q. Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, pages 3319-3328. PMLR, 2017.
J. Tiedemann and S. Thottingal. OPUS-MT- Building open translation services for the World. In Proceedings the 22nd Annual Conferenec of the European Association for Machine Translation (EAMT), Lisbon, Portugal, 2020.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Guyon, U. von Luxburg, S. Bengio, M. Wallach, R. Fergus, V. N. Vishwanathan, and R. Gamett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998 6008, 2017. URL http:: WwK.tttppp//prrcceeiingg. neur cc/ paper
Abst html
J. Vig. A multiscale visualization of attention in the transformer model. arXiv:1906.05714 2019.
arXiv preprint
Z. J. Wang, R. Turko, and D. H. Chau. Dodrio: Exploring transformer models with interactive visualization. arXiv preprint 1422v22103.14625, 2021.
M. L. Waskom. seaborn: statistical data visualization. Journal of Open Source Software, 6060),302,, 2021. doi: 10.21 (0.21105/joss.0002 03021. URL httttt:://abii.r///1/00 2tt.://0...11////0/01 03021
12

---

Under review as a conference paper at ICLR 2023
T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al. Huggingface's 's transformers: State-of-the of-the-ataete art natural language processing. arXiv preprint arXh::1910.037700 2019.
Y. Zhang, S. Sun, M. Galley, Y. Chen, C. Brockett, X. Gao, Gao, J. Liu, and B. Dolan. DIALOGPT : Large-scale generative pre-training for conversational response generation. In A. Celikyilmaz and Wen, editors, Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics: System Demonstrations, ACL 2020, Online, July 5-10, 2020, pages 270-278. Association for Computational Linguistics, 2020. doi: 10.18653/v 1tt3////2222...mmm acl-demos.30. URL
https::/do. org/lo. 1tt.e11163//////0 acl demos.... 30
13

---

Under review as a conference paper at ICLR 2023
A SHAPLEY VALUE PROPERTIES
1. Efficiency. All of the available payoff u(P) is distributed between the players: u(P) EiEp (P)
2. Symmetry.
Two players that have the same impact on the total payoff when joining coalition, receive the same share of the payoff: AS _ P((i,). v(SU(i)) v(S
sp,(P,) v) (P))).
3. Null Player. A player that has zero impact upon joining coalition, receives no share the total payoff: VS _ MM.. v(S) v(S U {i}) p.(P, u) = 0.
4. Additivity. The share of player in TU game (P. v w) is the sum of their shares in games (P,v) and w): vi E P. (P,, w) (P, (P.
B HEAD TASK ANALYSIS: LTL UNTIL-OPERATOR
In this experiment, provide another LTL example where one of the heads is focusing the temporal operator the formula and another focusing solely on the propositions of the formula (see The input formula is: all 1 where 1'l denotes that finally must occur. The
Figure 7 network correctly outputs the following trace: trace a A b; {1}.
&
0.2269
0.1769
U
0.2500
U
0.1806
0.2216
0.2032
b
0.2262
b
0.2500
0.2500
0.2161
0.2151
0.2500
0.2500
0.2500
attention flow encoder
attention flow encoder
Figure 7: Heatmap the attention flow for two heads: The left head focuses on the until-operator and the right head focuses the propositions.
C ADDITIONAL FIGURES
C.I HEATMAPS
Single Head Attention Flow in RoBERTa. F depicts the attention flow the first head in the RoBERTa model. Intuitively, the word killer dominates the sentiment of the sentence. However, the output of RoBERTa a positive sentiment, although the attention flow mainly on the word killer (see Figure 10). Analyzing the individual heads, with its maximal value (1.0), which could be one explanation for the output of the model.
observe that head 0 attends the smiley
Bias in DialogPT. Figure 9 shows the attention flow from each token to the current output. While we observe slight changes of the computed attention flow for each token, the first input token The highly attended, more than two times the attention flow than any other token. Note that this observation does not directly translate into a bias in the model, solely shows that the distribution of attention is biased.
14

---

Under review as a conference paper at ICLR 2023
1.0000
John
1.0000
is 1.0000
a
0.6850
good
0.4998
killer
0.6968
1.0000
0.4181
0.5323


0.2874
attention flow
Figure Heatmap showing head of RoBERTa for the example Figure
C.2 FLOW NETWORKS
Encoder Only. Figure 11 shows the flow network for RoBERTa. The underlying architecture is encoder only with 12 layers, represented by the 12 layers of attention edges between the nodes, and an input sentence with 10 tokens, represented on the y-axis of the network. The special property of RoBERTa the classification token at position 0 only the attention flow this token in the last node layer is important.
Encoder Decoder. Figure 12 shows the flow network for OPUS-MTTEN-DE. The underlying architecture consists of an encoder and a decoder with layers each, connected by the cross attention edges in between. For each input token and auto regressive token, we compute the attention flow to each predicted token. In Figure the attention flow for the third predicted token is computed.
Decoder Only. Figure 13 shows the flow network for GPT-2 with the underlying decoder only architecture. The model has 12 layers, attention can only flow from previous auto regressive tokens, including the input tokens. We start computing attention flow for the first output token, which is connected the terminal node in Figure 13
15

---

Under review as a conference paper at ICLR 2023
The 0.9412
brother
0.3908
of 0.4005
the 0.4078
mother
0.4135
went
0.4058
shopping
0.3515
0.3476
Who
0.3264
went
0.3371
shopping
0.3595
0.3727


0.304
The
0.1528
brother
0.1033
of
0.0799
attention flow
Figure 9; Heatmap showing the bias towards the first token in DialogPT.
12.0000
12.0000
15 John <50
12.0000
3.5416
John
3.7901
4.5775
4.9453
5.2016
5,5566
3.2202
3.4445
3.1044
good
5,1680
killr
6.8556
kiler good
4.8853
killer
7.9907
12.0000
12,0000
9.1259
12.0000

>
12.0000
3,2456
attention fow
atlention flow
3.7156
12,0000
12.0000
attention flow
Figure 10: Heatmap showing the attention flow for 3 variations of the same sentence in RoBERTa
16

---

Under review as a conference paper at ICLR 2023
Figure 11: The flow network the encoder-only network RoBERTa for the example in Figure
Figure 12: The flow network of the encoder decoder architecture OPUS-MT-EN-DE for the input "The father cooked dinner." and the predicted tokens "Der Vater kochte Abendessen".
17

---

Under review as a conference paper at ICLR 2023
Figure 13: The flow network for the decoder only architecture GPT-2 for the example in Figure 3a
18


Reviews:
[{"review_id": "xnfKeF5kIg9", "reviewer": "Reviewer_4MwZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper constructs an attention flow network out of encoder (& decoder) Transformers, and release a library to visualize the attention flow of Transformers.", "summary_of_the_review": "Overall, the paper feels like an extension to the original Attention Flows paper and not like a standalone paper. The experiments are limited, and the knowledge uncovered by the visualizations and methods are not impactful.\n\nI did not read the Appendices thoroughly.", "strength_and_weaknesses": "**Strengths**\n\n- (+) Authors explain that Attention Flow can be connected between an encoder and decoder of a Transformer.\n\n**Weaknesses**\n\n- (- -) The paper’s analysis is very limited. <5 sentences are analyzed in the main paper to show that their method works. The insights they uncover from the shapley values about the role that individual heads play in transformer attention are not novel.\n- (- -) The visualization library seems limited and not thoroughly introduced even though it was a central part of the abstract. The paper’s figures simply show tokens next to shapley values as colored blocks. The background color of the tokens overemphasizes small differences in the shapley value when all values are similar and is more deceiving than helpful. Additionally, the fact that attention flow primarily attends to punctuation and special tokens is not meaningful to understanding the data domain.\n- (-) Across the board, I am unable to understand why the experimental results are particularly meaningful. The experimental section feels more like showing that you can get numbers at all using this method.\n\n **Other Comments**\n\n- Statement “In practice, however, heads are biased towards keeping their respective tasks” [across layers] is not defended, and it has been my experience that in Transformers this is entirely not the case. In Transformers that have no weight sharing, there is no mechanism that encourages a head to perform the same association function across layers.\n- Figs 4 and 5 are not self contained. I cannot see the input tokens in the plot itself, making the results difficult to interpret.\n- Figure 2’s explanation on the top of page 4, “encoder (top) and a decoder (bottom)” is inconsistent with the design of the figure itself.\n- Unclear what “positional independence” (sec 1 page 2) of computed maxflow values means — positional encoding is embedded into the token representation itself and attention learns to attend to each token+position? I could not find where this was further explained in the paper", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "l3FClHE8xgf", "reviewer": "Reviewer_uJJx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method to quantify the impact of each input token on the outcome of a Transformer model. The algorithm constructs a flow network whose edges contain attention values, and solves the max flow problem for each vertex corresponding to an input token. The idea is applied to encoder only, decoder only, and encoder-decoder models.\n\nThe paper makes some connections with game theoretical concepts to justify the meaningfulness of the approach and the computed scores.\n\nFinally, the paper shows some experiments where the token-importance values are computed, and offers an explanation regarding the underlying logic or intuition behind each example.", "summary_of_the_review": "This paper extends [1, 2] to decoder only and encoder-decoder architectures, and provides some experiments where the method is applied. The contribution of the paper seems modest, and maybe some of the mathematical formalization could be replaced with more examples or applications.\n\n[1] = Abnar and Zuidema, Quantifying Attention Flow in Transformers.\n[2] = Ethayarajh and Jurafsky, Attention Flows are Shapley Value Explanations.", "strength_and_weaknesses": "The overall approach makes intuitive sense (note it isn't new), but the outputs are numbers that are still hard to interpret or use, as there are many moving parts. After going over the paper, I ended up with the impression of having read about a technical tool that isn't solving any specific problem. Given the amount of follow-up work on [1], it seems the ML community probably has found applications for it, though.\n\nI appreciate the authors' efforts and honesty at the end of the paper to list limitations of the work. One example is the fact that Transformer models also apply layers other than attention, like MLPs. I'm not sure if the proposed method accounts for token transformations via those; maybe the fact that they affect the subsequent attention scores is enough? i.e. if an MLP layer zeroes-out a token, the attention scores used to solve the max-flow problem (matrix A in the paper) will propagate the zeros, right? Otherwise results would be misleading.\n\nIn terms of experiments, as a proposal, I think a simple vision example should be much more informative. Each input token in VIT corresponds to a patch, and for image classification, the authors could show the input image superimposed with the corresponding attention score of each patch. Ideally, the attention centers around the object that is to be classified. It would also be interesting to see how solving a max-flow problem with hundreds of input tokens (like in VIT) scales in terms of runtime.\n\nThe authors offer some time estimates in 4.1 (text completion), but given the complexity and engineering required to compute attention flows, I was expecting a more in-depth analysis of how expensive it is to compute these numbers for real-world models. Grouping inputs or skipping layers, as mentioned in Section 2.5 is an interesting approach, but no evidence is provided on that direction suggesting it could work.\n\nIt's hard to conclude much from some of the experiments (like Figure 4 or 5a).\n\n[1] = Abnar and Zuidema, Quantifying Attention Flow in Transformers.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gn61_AxpSR", "reviewer": "Reviewer_DmXz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the author provides a comprehensive formulation of the attention flows for different architectures of transformers (encoder, decoder, encoder-decoder). In order to account for the auto-regressive decoder structure, the author adjusted the attention flow to ensure the positional independence of the computed maxflow values. Following the previous work, the author further shows that the maximum attention flow corresponds to the Shapley value under the three architectures of transformers. ", "summary_of_the_review": "Explaining the decision from large language models has always been a focus since the popularity of these models. This paper improves on top of similar existing works but providing a comprehensive formulation on the attention flow that covers decoder and encode-decoder too. It is of great interest to this community to see progress that leads to better explainability of the model. On the other hand, this paper's main contribution is not significant enough for ICLR since the major works (both the attention flow and the equivalence to Shapley value) exist before and it did not contribute significantly outside the existing framework. The performance evaluation is also tricky since only a few examples are shown but lacks of more experiments to demonstrate the strength of attention flow method. I would like to see some improvement on the matter. ", "strength_and_weaknesses": "The strength of this paper lies in the following aspects:\n\n1. Although the attention flow formulation for transformer encoder has been proposed before, this paper extends this to cover the rest of two architectures so that this method can be applied to auto-regressive structure. The model formulation is solid with minor typos. \n\n2. The connections between attention flow values and the Shapley values has been shown before. This work, however, is able to explore the assumption on the positional independence so that the proof is simplified and is able to avoid some issues behind the underlying assumptions of the previous proof. \n\nThe weakness of this paper lies in the following aspects:\n\n1. Lack of objective and reliable metric to decide if the attention flow is relevant or not. It is stated in the paper also that the attention flow value is not supposed to be interpreted as causal factors to the prediction or translation. It is merely an associational factor. This together with lack of large scale studies and peer-reviews in terms of the performance of the method limits its application in real life decision making. \n\n2. The proof that the attention flow is Shapley value relies on the effectiveness of the positional independence assumption. A solid verification and analysis is necessary to convince the reader that the positional independence is guaranteed in experiments. It is very challenging in general to check on this assumption even if the weight is normalized for tokens output later.\n\n3. The proof that the attention flow is Shapley value under the positional independence assumption needs to be shown in detail instead of explained vaguely. The contribution of this paper is its mathematical formulation of the attention flow. It is necessary to show this critical proposition. Hope that this part can be added in appendix later on.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bH59bXG0lA", "reviewer": "Reviewer_wsJr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work extends the attention flow method proposed in Abnar and Zuidema 2020 to encoder-decoder and decoder-only transformers. The major contribution is based on the observation that later predicted words have more incoming edges than earlier words, such that to ensure positional independence, this work proposes a method to normalize maxflow values. In addition, this work draws connection between maxflow attention and Shapley values by defining payoffs as the sum of maxflows and showing the equivalence under this definition. Experiments on several tasks show that this method is able to gain insights into token importance for a prediction task.", "summary_of_the_review": "My major concerns are: 1. the change to the original attention flow paper is incremental; and 2. a correlation study with feature importance found by other methods (such as Shapley values) is missing and it's not clear if the found maxflow values mean anything. Therefore, I'm leaning towards rejecting this paper.", "strength_and_weaknesses": "Strengths:\n1. Extends attention flow to encoder-decoder and decoder-only transformers.\n2. Empirical analysis of token importance on several tasks.\n\nWeaknesses:\n1. The change to Abnar and Zuidema 2020 seems incremental.\n2. The connection to Shapley values is drawn by defining value function to be based on maxflows, so the equivalence is not surprising at all. A more interesting connection would be to use the actual log probability of the target token as payoff values and see if there's a correlation.\n3. Related to 2, it is not clear if attention maxflows reflect actual token importance. It would be more convincing if the maxflow values can be compared against feature importance obtained using other methods (such as Shapley values using log probability of target token as payoffs, or simply gradient-based saliency maps). For example, does the first-token bias found by maxflow hold for Shapley values?\n4. The analysis on individual attention heads doesn't make that much sense to me, since the same head id across different layers does not mean they have anything in common.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject