Analyzing Feed-Forward Blocks in Transformers through the Lens of Attention Maps
Authors: Goro Kobayashi
Tatsuki Kuribayashi
Sho Yokoi
Kentaro Inui
Venue: ICLR 2024 spotlight (2024)
Keywords: Transformer
Attention map
Feed-forward
Contextualization
Interpretation
Analysis
Pre-trained models
Masked language models
Causal language models
Primary area: visualization or interpretation of learned representations
TL;DR: We analyze the input contextualization effects of Feed-Forward blocks in Transformer-based models by rendering them in the attention maps.

Abstract: Transformers are ubiquitous in wide tasks.
Interpreting their internals is a pivotal goal. 
Nevertheless, their particular components, feed-forward (FF) blocks, have typically been less analyzed despite their substantial parameter amounts.
We analyze the input contextualization effects of FF blocks by rendering them in the attention maps as a human-friendly visualization scheme.
Our experiments with both masked- and causal-language models reveal that FF networks modify the input contextualization to emphasize specific types of linguistic compositions. 
In addition, FF and its surrounding components tend to cancel out each other's effects, suggesting potential redundancy in the processing of the Transformer layer.

Published as a conference paper at ICLR 2024
ANALYZING FEED-FORWARD BLOCKS IN TRANS- FORMERS THROUGH THE LENS OF ATTENTION MAPS
Goro Kobayashi 1.3 Tatsuki Kuribayashi Sho Yokoi Kentaro 2nui 'Tohoku University, "MBZUAI, "RIKEN
goro. toh.ku.cchhhccccco tatsuki. www.kkkuiiiiaainnnaaa ac. ae yokoi@t:choku.ac.p kentaro. inni manimmmzuai..om ac ae
ABSTRACT
Transformers ubiquitous in wide tasks. Interpreting their internals is pivotal goal. Nevertheless, their particular components, feed-forward (FF) blocks, have typically been less analyzed despite their substantial parameter amounts. We alyze the input contextualization effects of FF blocks by rendering them in the attention maps human-friendll visualization scheme. Our experiments with both masked- and causal-langaage models reveal that FF networks modify the in- put contextualization emphasize specific types of linguistic compositions. addition, FF and its surrounding components tend to cancel out each other's ef- fects, suggesting potential redundancy the processing of the Transformer layer.
yys
INTRODUCTION
Transformer composed of several components (e.g., self-attention and feed-forward networks); tracking and interpreting its component-by-compee enrich the mechanistic interpretation of the model's inner workings (Kobayashi et al. 2020 2021] A. for more works). One straightforward yet popular approach is to render vanilla attention weights, reflecting how strongly a particular context token contributes computing an output representation ($ 2). and identify which attention head a specific type of contextualization is performed in Transformer (Clark et al. [2019] Kovaleva et al. 2019). Re- cent studies have extended this approach by incorporating more components beyond Query-Key matrix multiplication, e.g., residual connections, enabling to analyze how input-contextualization shaped nar & Zuidema 2020; Kobayashi et al. 2020 2021: Modarressi et al. 2022] see $ AA.2 for more works). Such an interpretation scheme has several advantages: (i) Attention mechanisms are spread out in the entire Transformer architecture; thus, if one view their surrounding component's processing through this "semi-transparent window," these pictures can complete the Transformer's internal processing, (ii) token-to-token relationships miliar to humans than directly observing high-dimensional, continuous intermediate representa- tions/parameters, and (iii) input attribution is of major interest in explaining the model prediction, and (iv) such attention map refinement approaches tend to estimate better attributions than, c.g.,
intermediate processing have been aimed to
Modarressi al. 2022;
through the lens of attention maps (Brunner al. 2020, Ab-
entire component-yy-componee
more fa-
gradient-based methods (Modarressi al. 2022 2023).
Nevertheless, in this attention-map refinement approach, FF networks have typically been over- looked from the analysis scope, although there are several motivations consider FFs. For example, FFs account for about two-thirds of the layer parameters in typical Transformer -assed models, such as BERT and GPT series; this implies that FF has the expressive power to dominate the model's inner workings. In addition, there is growing general interest in FFs with the rise of FF-focused meth- ods such as adapters (Houlsby et al. 2019), although analyzing these specially designed models is beyond this paper's focus. Furthermore, it has been reported that FF's indeed perform some linguis- tie operations, while existing studies have not explicitly focused on input contextualization (Geval et al. 202 Dai et al. 2022). For example, Oba al. (2021) explored the relationship between FF's neuron activation and specific phrases; Geva et al. (2021), Meng et al. (2022), and Dai et al. (2022) examined the knowledge stored in FF's parameters through viewing FF key-value memonis.

---

Published as a conference paper at ICLR 2024
In this study, analyze the FF blocks in the Transformer layer, i.e., FF networks and their sur- rounding residual and normalization layers, with respect to their impact on inpur contextualization through the lens of attention maps. Notably, although the FFs are applied to each imput representa- tion independently, their transformation can inherently affect the input contextualization ($ 3 and our experiments show that this indeed occurs ($ D. Technically, we propose a method to compute attention maps reflecting the FF blocks' processing by extending a norm-based analysis (Kobayashi al. 2020, 2021), which has several advantages: the impact of input ([]==]) is considered unlike the vanilla gradient, and that only the forward computation is required. Although the original norm- based approach can not be simply applied to the non-linear part in the FF, this study handles this limitation by partially applying an integrated gradient (Sundararajan et al. (017) and enables us to track the input contextualization from the information geometric perspectives.
Our experiments with both masked- and causal-language models (LMs) disclosed the contextual- ization effects of the FF blocks. Specifically, we first reveal that FF and layer normalization in specific layers tend to largely control contextualization. We also observe typical FF effect pat- terns, independent of the LM types, such as amplifying a specific type of lexical composition, e.g., howwords-t-woodd and first observe that the FF's effects are weakened by surrounding residual and normalization layers ($ (6)) suggesting redundancy in the Transformer's internal processing.
www.-tomi--wood--pppeeeee--eeeee word-expression constructions ($ 5.2). Furthermore, also
2 BACKGROUND
Notation: Boldface letters such as æ denote row vectors.
Transformer layer: The Transformer architecture (Vaswani et al. 2017) consists of series of layers, each of which updates each token representation E; E IR" in input sequence X = ......., x., ]T E Rexd
new representation y: Rd. That is, the information of context X is added to F; and æ; is updated to yı. We call this process contextualization of x;, Each layer is composed of four parts: multi-head attention (ATTN), feed-forward network (FF), residual connection (RES), and layer normalization (LN) (see Fig. [T. Note that we use the Post-LN architec- ture (Vaswani et al. (017)) for the following explanations, but our methods can simply be extended to the Pre-LN variant (Xiong et al.
2020). A single layer can be written as a composite function:
y: Layor(a;;
(FFB ATBB (2;:X)= = 2LN2 o RES2 o FFo LN1 o RES1 o ATTN)
We call (LN1 RES1 o (-) attention block (ATB), and (LN2 o RES2 o (-) feed-forward
block (FFB). Each component updates the representation as follows:
ATTN(&;; X)= = (En E, at Wr + bo) + bo E Rd
(1)
where o = (z, We (2) WK + +x) RR
(2)
FF(2;) = g(z Wi + b1) W; + b2 E Rd
(3)
(RES (z:) -(z)) + 2, Edd
(4)
LN(2:) = 2: -m(zi) 3 7 + 3
(5)
s(,,
where W.y denote weight parameters, and b, 3 denote bias parameters corresponding to query
(Q), key (K), value (V), etc. f: RR + Rd g: TRdd' - TRd m: Rdd - IR, and 8: Rd - R denote an arbitrary vactor-valued function, activation function in FF (e.g., GELU, Hendryykk & Gimpel, 2016), element-wisg mean, and element-wise standard deviation, respectively. h denotes the head number in the multi-head attention. See the original paper (Vaswani et al. 2017) for more details.
Attention map: Our interest lies in how strongly each input æy contributes to the computation of a particular output y4. This is typically visualized as n x n attention map, where the (i, j) cell represents how strongly æj contributed to compute y,. A typical approximation of such a map is facilitated with the attention weights (a or Σn, att henceforth denoted as,;) (Clark et al. 2019 Kovaleva al.) 2019;; however, this only reflects specific process (QK attention computation) of the Transformer layer (Brunner et al. 2020 Kobayashi et al. 2020 Abnar & Zuidema 2020).
Looking into Transformer layer through attention map: Nevertheless, attention map is not only for visualizing the attention weights; existing studies (Kobayashi et al. 2220;; (02T) have an- alyzed other components through the lens of refined attention map. Specifically, Kobayashi et al.
2

---

broader component of the Transformer Published as a conference paper at ICLR 2024
Post-LN Transformer layer
Pre-LN Transformer layer
Output vectors Ya 2 Y/
Output vectors Y4 Y5
ATBEFRESLN (Ours)
Layer Nomtalization (LN2)
ATBLNFFRES (Owrs)
Residual
Connection (RFS2)
Residial Connection (RES2)
Forward Network (FIP)
ATBFERES (Ours)
ATBLNFE (Ours)
Feod-Forennt Forward Network (FF)
Layer Normulization (LN2)
ATBFF (Ours)
Feud-Forwand
ATBLN
Feud-Forwand (FFB)
(Ours)
(FFB)
Layer Normalization (LNI)
Residual Connoction (RESI)
Residual Connection (RES1)
ATB (Kobayasii 21)
(Kobayashi '21)
Multi dead Attention (ATTN)
Pouuf Multi bead Attention (ATTN)
Attention weights
Attention Block (ATR)
Attention weights
Layer Normalizztion
(LNNI)
Attention
Block
(ATB)
WyMgoryuWWs Token-to-token attention map
X;
Input vectors
Input vectors
Figure 1: Overview of the Transformer layer for Post-LN and Pre-LN architectures, annotated with analysis scopes, e.g., ATBFFRESLN. The right part of this figure (token-to-token attention map) il- lustrates the ceapponent-by-comooeee changes of the attention maps. See Appendix B for concrete examples of attention maps.
(2020) pointed out that the processing in the multi-head QKV attention mechanism can be written sum of transformed vectors:
NTTIN =E
F FATTN
Zz: X) + BATTN
where FATTN (zy; X) = of (2,, W BATTN = by Wo + bo-
(7)
Here, input representation 3; updated to y ATTN by aggregating transformed inputs FATIN (æy; X). Then, |FATTNN (æz;; X)II instead of aij alone is regarded refined attention weight with the simple intuition that larger input contributes to output more in summation. This refined weight reflects not only the original attention weight ass.j. but also the effect of surrounding processing involved in F, such as value vector transformation. We call this strategy of decomposing the process into the sum of transformed vectors and measuring their norms norm-based analysis, and the obtained
attribution matrix is generally called attention map in this study.
This porm-based analysis has been further generalized layer. Specifically, Kobayashi al. (202T) showed that the operation of attention block (ATB) could also be rewritten into the sum of transformed inputs and a bias term:
ATB y
ATB(,;; X) = E, FATB (ar;; X) + AATB
(8)
Then, the norm |FABB (ay; X)I| was analyzed to quantify how much the input æ, impacted the computation of the output y ATB through the ATBs. See Appendix C for details of FATB and ATTB
PROPOSAL: ANALYZING FFBs THROUGH REFINED ATTENTION MAP
The transformer layer is not only ATB; it consists of an ATB and FFB (Figs. I and 2 Thus, this study broadens the analysis scope to include the entire FF block consisting of ford-rorward. residual, and normalization layers, in addition the ATB. Note that FFBs do not involve token- wise interaction among the inputs X; thus, the layer output y, Layer can be written transformed ATB Then, our aim is to decompose the entire layer processing as follows:
y Layer FFB E, FATB (zy; X) + BTTB)
(9)
E, F. (ay; X) Layer
(10)
3

---

Published as a conference paper at ICLR 2024
Here, the norm || (z;; [| can render the up- dated attention map reflecting the processing in the
Y
Layer
Layer
y
entire layer. This attention map computation also be performed component-y-.compoeent we
player
(x3)
player (Xxs)
can track each FFB component's s effect.
Feed Forward Block (FFB)
y
Why FFBs can control contextualization pat- terns: While FFBs are applied independently each input representation, FFB's 's each input already contains mixed information from multiple token rep- resentations due to the ATB's 's process, and the FFBs can freely modify these weights through a nonlinear transformation (see Fig. 2
PATB (x5)
pATB (xx)
Attention Block (ATR)
X2
X3
An illustration of possible con- textualization effects by FFB. The FFB does not have the function of mixing input to- kens together; however, its input already con- tain mixed information from multiple tokens, and the FFB is capable of altering these weights. Here, output y/3 computed based on [01,-.. .zs]. Based on the vector norm, (11) the most influential input seems to be æt be- fore FF; however, after the FF's transforma- tion, 2y becomes the most influential input.
Figure 2:
Difficulty to incorporate FF: While FFBs have the potential to significantly impact contextualiza- tion patterns, incorporating the FF component into norm-based analysis is challenging due to its non- linear activation function, g (Eq. 3). which cannot be decomposed additively by the distributive law:
F(z))) * Eg (xy))
This inequality matters in the transformation from Eq. to Eq. [0] That is, simply measuring the norm li (rr is not mathematically valid. Indeed, the FF component has been excluded from previous norm-based analyses. Note that the other com- ponents in FFBs, residual connection and layer normalization, can be analyzed in the same way proposed in Kobayashi et al. (2021).
Integrated Gradients (IG): The IG (Sundararajan et al. 2017) is a technique for interpreting deep learning models by using integral and gradient calculations. It measures the contribution of each input feature to the output of neural model. Given function f: R" + R and certain input L (af,..., 22)) E R", IG calculates the contribution (attribution) score (E R) of each input feature oyf E IR to the output f(x') E R::
0s Ja= 0z, ln b+ ofx" b)
f(x') = Σ TT, + s(b). IG((a'; = (x!,- - b.)
do. (12)
Here, b E R denotes a baseline vector used to estimate the contribution. At least in this study, it is set to zero vector, which makes zero output when given into the activation function (0( = 0), to satisfy desirable property for the decomposition (see Appendix D.2 for details).
Expansion to FF: We explain how use IG for the decomposition of FF output. As aforemen- tioned, the problem lies in the nonlinear part, thus we focus on the decomposition around the non- linear activation. Let us define F.tee (xx, := (xx;; X X)W1 E gd as the decomposed vector prior to the nonlinear activation g: Rdd Rd The activated token representation y' E Rd' is written as follows:
yl = 9 Eラ= F. (0x)) EE= h (F. Pre (xI),..... P. (xx): 0),
(13)
TG3 (F:)(()=))
Pre ( (m.))
h2 (F. Pro 9 (x1).....
Pro (x, 0)
(14)
IG, 9 E (x,,)
[d']: (d'))'d)):
where the transformation T: 1 + R is defined as g(11 Rn) = g + + 2,). which adds up the inputs into a single scalar and then applies the element-level activation g: R + R. The function hy: Ro + Rd yields how strongly a particular input F.re (ax,) contributed to the output v/- Each element hy [k] indicates the contribution of the k-th element of input P. (x,)
4

---

Published as a conference paper at ICLR 2024
to the k-th element of output kk] Note that contribution can be calculated element- because the activation is applied independently to each element.
Notably, the sum of contributions across inputs matches the output (Eq. 13 achieved by the de- sirable property of IG completeenss (Sundararajan et al.] (2017). This should be satisfied in the norm-based analysis and ensures that the output vector and the sum of decomposed vectors the same. The norm || h, (F."h ((x)).... F. (xnn); 0) is interpreted as the contribution of input 2. to the output y of the activation in FF
Expansion to entire layer: Expanding this, the contribution of layer input æ; to the i-th FF output is exactly calculated as ||,,, (F, (xx ..., F.Pee (.,, g. 0) W2ll- Then, combined with the exact decomposition of ATTN, RES, and LN shown by Kobayashi et al. (2020; 202T), the entire Transformer layer can be written as the sum of vector-valued functions with each input vector æ, E
X as argument written in Eq. [O] This allows us render the updated attention map reflecting the processing in the entire layer by measuring the norm of each decomposed vector.
4 GENERAL EXPERIMENTAL SETTINGS
Estimating refined attention map: To elucidate the input contextualization effect of each compo-
nent in FFBs, we computed attention maps by each of the following four scopes (Fig. D
ATB (Kobayashi al. 2021 Analyzing the attention block (i.e., ATTN, RES1, and LNI)
using vector norms as introduced in Eq. g
AFBFF (proposed): Analyzing components up FF using vector norms and IG.
ATBFFRES (proposed): Analyzing components up to RES2 using vector norms and IG. ATBFFRESLN (proposed): Analyzing the whole layer (all components) using vector norms and IG.
We will compare the attention maps from different scopes to separately analyze the contextualization effect. Note that if the model adopts the Pre-LN architecture, the scopes will be expanded from ATB to ATBLN, ATBLNFF, and ATBLNFFRES (see the Pre-LN part in Fig. I
Models:
We analyzed 11 variants of the masked language models: six BERTs (uncased) with dif- ferent sizes (large, base, medium, small, mini, and tiny) (Devlin et al.
2019; Ture et al. 2019), three BERTs-base with different seeds (Sellam et al.] (002). plus two RoBERTas with different sizes (large and base) (Liu et al.) 2019). We also analyzed two causal language models: GPT-2 with 117M parameters and OPT (Zhang et al. 2022) with 125M parameters. Note that the masked language models adopt the Post-LN architecture, and the causal language models adopts the Pre-LN architecture.
Data: We used two datasets with different domains: Wikipedia excerpts (992 sequences) (Clark et al. 2019 and the Stanford Sentiment Treebank v2 dataset (872 sequences from validation set) (SST-2, Socher et al., 2013). The input was segmented by each model's pre-trained tokeniz- ers; analysis was conducted at the subword level
5 EXPERIMENT 1: CONTEXTUALIZATION CHANGE
Does each component in FFBs indeed modify the token-to-token contextualization? We analyze the contextualization change through each component in FFBs.
5.1 MACRO CONTEXTUALIZATION CHIANGE
Given two analysis scopes (e.g.. before and after FF; ATB + ATBFF), their contextualization pattern change was quantified following some procedures of representational similarity analysis (Kriegeskorte et al. 2008). Formally, given an input sequence of length n, two different attention maps from the two scopes are obtained. Then, each attention map (R was flattened into a vector (IR") ), and the Spearman's 's rank correlation coefficient, p between
Calculating contextualization change:
https: : / i thub. com/cit rkkev wwt.kket/attttttt..eeeeea on-onnnn ana lom-anllysaa lysis
For masked language models, each sequence was fed into the models with masking 12% tokens as done in the case of the BERT training procedure.
5

---

Published as a conference paper at ICLR 2024
the two vectors were caleulated. We report the average contextualization change p across input sequences. We will report the results of the BERT-base and GPT-2 on the Wikipedia dataset in this section; other models (including OPT) and datasets also yielded similar results (see Appendix G.I).
The contextualization changes through each component in FFBs are shown in Fig. 3 A higher score indicates that the targeted component more drastically updates the contextualization patterns. Note that we explicitly distinguish the pre- and post-layer normalization (PRELN and POSTLN) in this section, and the component order in Fig. 3 is the same as the corresponding layer architecture.
2.00
1.00
1.00
BERT (PostLN) LN)
Change
0.75
0.75
0.7
0.50
0.25
255
0.00
0.00
0.00
1011122Ag.
101112
Avg.
Layer
Layer
Layer
(a) +FF
(b) +RES
(c) +LN
(ATB ATBFF)
(ATBFF ATBFERES)
(ATBFFRES ATBFERESLN)
GPT--2 (PreLN)
Change
0.75
0.75
0.50
0,50
0.50
0.25
0.25-
0.25
0.00
0.00
101112
Aug.
101112 Avg.
Layer
Layer
Layer
(d) +LN
(c) +FF (ATBLN + ATBLNFF)
(1) +RES
(ATB + ATBLN)
(ATBLNFF + ATBLNFFRES)
Figure 3: Contextualization changes between before and after each component in FFBs (FF, RES2,
and LN2) of BERT and GPT-2. The higher the bar, the more drastically the token-to-token contex- tualization (attention maps) changes due to the target component.
We generally observed that each component did modify the input contextualization. especially in particular layers. For example, in the BERT-base, the FF in 3rd and 9th-11th layers and normaliza- tion in 10th-11th layers incurred relatively large contextualization change. Comparing the BERT- base and GPT-2, the by-layer average of contextualization change by FF and LN was similar across these LMs: 0.21 and 0.15 for FF BERT and GPT-2 and 0.27 and 0.38 for LN in BERT and GPT-2, respectively. In contrast, there seem mainly two differences between BERT and GPT-2: (i) the effect of RES, and (ii) the layers at which the high-impact FF's are located. At least for the FFs of GPT-2, the shallow layer's high impact is consistent with some existing studies (Meng et al.) 2022 Geva jct al., 2023] Gromov et al. 2024), but revealing the cause of such gaps between BERT and GPT-2 would be future work. Note that Figs. 14 to 18 in Appendix G show the results for other model variants, yielding somewhat consistent trends that FFs and LNS in particular layers especially incur contextualization changes.
5.2 LINGUISTIC PATTERNS FF'S CONTEXTUALIZATION EFFECTS
The FF network is a completely new scope in the norm- based analysis, while residual and normal- ization layers have been analyzed in Kobayashi et al. (2021) Modarressi et al. (2022). Thus, we fur- ther investigate how FF modified contextualization, using BERT-base and GPT-2 on the Wikipedia dataset as case study.
We compared the attention maps before and after FF, Specifically, we subtract a pre-FF attention map from a post-FF map (Fig. 4b: we call the resulting diff-map FF- amp matrix and the values in each cell FF-amp score A larger FF-amp score of the (i, j) cell
Micro contextualization change:
Before the subtraction, the two maps were normalized so that the sum of the values of each column was 1; this normalization facilitates the inter-method comparison.

---

Published as a conference paper at ICLR 2024
presents that the contribution of input æ output y: is more amplified by FF. Our question is which kind of to- ken pairs gain a higher FF-amp score
FF-amp matrix
ATBFE
ATB
FFs amplity particular linguistic compositions: Based on the preliminary observations of high FF-amp token pairs, we set 7 linguistic categories typically amplified by FFBs. This includes, for example. token pairs consisting of the same words but different positions in the input. Ta- ble shows the top two token pairs w.r.t. the amp score in some layers and their token pair category. Fig. F Summmaizzes the ratio of each category in the top 50 token pairs with the highest FF-amp score Compared to the categories for randomly sampled token pairs (FR; the second rightmost bar in Fig. ()) and adjacent token pairs (AR; the rightmost bar Fig. 5). the amplified pairs in each layer have unique characteristics; for example, in the for- mer layers, subword pairs consisting the same token are highly contextualized by FFs. Note that this phenomenon of processing somewhat shallow morphological information in the former layers be consistent with the view of BERT pipeline perspective, with gradually more advanced processing from the former to the latter layers Tenney et al. (2019).
Figure 4: FF-amp matrix is computed by subtracting the attention map before FF (ATB) from that after FF (ATBFF).
Table 1: Token pairs sampled from the top 10 most- most-amplified subword pairs by FF in each layer of BERT (left) and GPT-2 (right). See Tables 2 and 3 in Appendix G.2 for the full pairs. The text colors are aligned with word pair categories, and the same colors are used Fig.
Layer amplified pairs by BERT's FF
Layer amplified pairs by GPT-2's FF
Subword
Compound
(opera, soap), (linnight, week)
(ies, stud), (ning, begin)
Common expression
3 (toys, #tfhop), C .t)
3
(than, rrather,, (Lothers, _among) (del, del), (_route, route)
Same token
6 (with, charged), (fleming, colin)
6
Named entity
9 (but, difficulty), (she, teacher)
8
(08, 22007, (_14, ,density)
Semanntcul connection
11 (tiny, tiny), (highway, highway)
12
Coperational, _not), (.daring, .has)
Others
subword
frequent expression
named entity
others
compound noun
same token
NNN
semantical connection
100
% pairs
pairs
8 BERT Layer
10
12
FR
AR
8
10
11
12
FR
AR
GPT-2 Layer
Figure 5: Breakdown of the category labels we manually assigned to top 50 pairs having the largest
FF-amp score each layer of BERT and GPT-2. We also assigned the labels to fully random 50 pairs ("FR") and adjacent random 50 pairs ("AR").
Simple word co-occurrence does not explain the FF's amplification: Do FF's simply amplify the interactions between frequently co-occurring words? We additionally investigated the relation- ship between the FF's amplification and word co-occurrence. Specifically, we calculated PMI for each subword pair on the Wikipedia dump and then calculated the Spearman's 's rank correlation coef- ficient between FF-amp scores each pair (w; wx) from BERT-base and PMI values We observed that the correlation scores were fairly low in any layer (coefficient values were 0.06 0.1)). Thus, found that the FF does not simply modify the contextualization based on the word co-occurrence.
4 We aggregated the average FF-amp score for each subword type pair (wi, wy). Pairs consisting the same position's 's token (w,, wx) and pairs nccurring only once in the dataset were excluded.
Onee af the authors of this paper has conducted the annotation.
We defined three types of co co-occureences in calculating PMI: the simultaneous occurrence of two subwords (i) in article, (ii) in a sentence, and (iii) in chunk of 512 tokens. No correlation was abserved for either type of co-occurrence. Pairs including special tokens and pairs consisting of the subword excluded.

---

Published as a conference paper at ICLR 2024
To sum up, these findings indicate that FF did amplify the contextualization aligned with particular types of linguistic compositions various granularity (i.e., word and NE phrase levels).
EXPERIMENT 2: DYNAMICS OF CONTEXTUALIZATION CHANGE
We analyze the relationship between the contextualization performed by FF and other components, such RES, given the previous observation that contextualization changes in particular compo- nent are sometimes overwritten by other components (Kobayashi et al. 2021). We will report the results of the BERT-base and GPT-2 on the Wikipedia dataset in this section; other models/datasets also yielded similar results (see Appendix H
+ FF
PF
RES
+FF RES
PF RBS LN (Same PostLN)
+FF RES LN (Next layer's PreL.)
1.00
1.00
Change
0.75
Chang
0.75
0.50
0.50
0.25
0.25
0.00
L Don
0.00
1
3
4
7 8
101111 12 Avg.
1
2
4
5
7 8 Layer
10 12 Avg.
Layer
(a) BERT
(b) GPT-2.
Figure Contextualization changes through FF, RES, and LN relative the contextualization performed before FF. The higher the bar, the more the contextualization changes.
Fig. O shows the contextualization change scores (1 described in $ 5.1 by FF and subsequent components: FF (+FF), FF and RES (FF+RES), and FF, RES and LN (FF+RES+LN), Note that analyzed the next-layer's LNI in the case of GPT2 (Pre-LN architecture) to analyze the BERTs and GPTs from the same perspective-whether the other components overwrite the comtextualiiation performed in the FF network. If a score is zero, the resulting contextualization map is the same as that computed before FF (after ATBs BERT or after LN2 in GPT-2). The notable point that through the FF and subsequent RES and LN, the score once becomes large but finally converges to be small; that is, the contextualization by FFs tends to canceled by following components. We look into this cancellation phenomenon with specific focus on each component.
6.1 FF AND RES
The residual connection bypasses the feed-forward layer as Eq. 4 Here, the interest lies how dominant the bypassed representation ær is relative to (ar)).. For example, if the representation ar has a much larger L2 norm than F'F()),, the final output through the RES2 o FF will be similar to the original input æ/; that is, the contextualization change performed in the FF would be diminished.
FF
400
Norm
40
RES2
Norm
RES2
200
20
2
3
10
12
2
4
5
10
Layer
Layer
(a) BERT.
(b) GPT-2.
Figure 7: Averaged norm of the output vectors from FF and the bypassed vectors via RES2, caleu- lated on the Wikipedia data for BERT and GPT-2.
RES2 adds a large vector: We observe that the vectors bypassed via RES2 are more than twice as large as output vectors from FF in the L2 norm in most layers (Fig. 7). That is, the representation (contextualization) updated by the FF tends to be overwriten/canceled by the original one. This observation consistent with that of RES1 (Kobayashi et al. 2021). Notably, this cancellation is
8

---

Published as a conference paper at ICLR 2024
weakened in the 10th-11tth layers in BERT and former layers in GPT2s, where FFss contextualiza- tion was relatively large (Figs. 3a and Be
6.2 FF AND LN
We also analyzed the relationship between the contextualization performed in FF and LN. Note that the layer normalization first normalizes the input representation, then multiplies a weight vector element-wise and adds a bias vector (Eq. 8).
Cancellation mechanism: Again, as shown in Fig. 6 the contextualization change after the LN (+FF+RES+LN) is much lower than in preceding scopes (+FF and +FF+RES). That is, the LNs substantially canceled out the contextualization performed in the preceding components of FF and RES. Then, we specifically tackle the question, how did LN cancel the FF's effect?
We first found that the FF output representation has outliers in some specific dimensions (green lines Fig. { and that the weight 7 LN tends to shrink these special dimensions (red lines in Fig. 8 In the layers where FF incurs relatively large impact coefficient between LN's 7 and mean absolute value of FF output by dimension was from -0.45 to 00.78 in BERTs and from -0. -0.22 to -0.59 in GPT-2 across layers. Thus, suspect that such specific outlier dimensions in the FF outputs encoded "flags" for potential contextualization change, and LN typically declines such FF's suggestions by erasing the outliers.
contextualizationn the Pearson correlation
Mean absolute value
30
2.0
Mean absolute value
10
20
LN wel we value
weight
1.0
-10
20
200
400
600
800
200
400
600
800
Dimension
Dimension
(a) 10th layer in BERT.
(b) 4th layer in GPT-2.
Figure 8: Mean absolute value in each dimension of the input/output vectors of FF across the Wikipedia data and the LN weight values the certain layer.
Indeed, observed that ignoring such special outlier dimensions (bottom 1% with the lowest value of 7) in calculating FF's contextualization makes the change score quite small; contextualization changes by FF went from 0.21 to 0.09 in BERT and from 0.15 to 0.02 in GPT-2 layer average. Thus, FF's contextualization effect is realized using very specific dimensions, and LN2 cancels the effect by shrinking these special dimensions. Note that related phenomenon was discovered by Modarressi et al. 2022): LN2 and LNI cancel each other out by outliers in their weights in BERT (see other related works in A.3 The observed cancellation mechanism over feed-forward and normalization layers also suggests redundant contextualization processing in Transformer AA.4) Further investigation is important future work.
CONCLUSIONS AND FUTURE WORK
We have analyzed the FF blocks w.r.t. input contextualization through the lens of a refined attention map by leveraging the existing nosm-based analysis and the integrated gradient method having an ideal property-ccompltenesss.
Our experiments using masked- and causal-language models have shown that FF's indeed modify the input contextualization by amplifying specific types of linguistic compositions (e.g., subword pairs forming one word). We have also found that FF and its sur- rounding components tend to cancel out each other's contextualization effects and clarified their mechanism, implying the redundancy of the processing within the Transformer layer. Applying our analysis to other model variants, such as Mistral (model with local attention, Jiang et al. (023) and Mixtral (model with mixture of experts, Jiang et al., 2024) will be our future work. addition, focusing on inter-layer contextualization dynamics could also be fascinating future directions.
"FFs in BERT's 3rd and 10th-11th layers and GPT-2's 1st-1 1th layers

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
We recognize that this study has not used any special data or methods with potential ethical issues. One recent issue in the whole community that neural-network-baseee models make predictions containing non-intended biases (e.g., gender bias). This paper gives a method for interpreting the inner workings of real-world machine learning models, which may help us understand such biased behaviors of the models in the future.
REPRODUCIBILITY
We have reported the used data and pre-trained models in the manuscript as much as possible, and no private or unofficial data people can not access are used. In addition, we used several variants of models, including different types of masked- and causa-LMs; this hopefully has con- tributed enhancing our results' generality (reproducibilitt). All the models can publicly avail- able through Huggingface's 's transformers framework (Wolf et al., 2020). Additionally, our analysis tool and experimental codes are publicly available at htttp:: githhu.. thub. com/ gotokoba6kk norm- analysiiiii of transformer
ACKNOWLEDGMENTS
We would like to thank the members of the Tohoku NLP Group for their insightful comments. This work was supported by ISPS KAKENHI Grant Number JP22/21492, JP22H05 106; JST CREST Grant Number JPMJCR20D2, Japan; and JST ACT-X Grant Number JPMJAX200S, Japan.
REFERENCES
Samira Abnar and Willem Zuidema. Quantifying Attention Flow in Transformers. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 4190-4197, 2020. doi:
m88............ URthttps: wttp:///wwwwwww..rr..o.oo.
anthology/2020.acc- 385
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Quantizable Transformers: Remov- ing Outliers by Helping Attention Heads Do Nothing. arXiv preprint, 2023. URL https:
7/axxiv. abs /20/6/2366..2 Tw..30...199999 12929v2
Gino Brunner, Yang Liu, Damián Pascual, Oliver Richter, Massimiliano Ciaramita, and Roger Wat- tenhofer. On Identifiability Transformers. In 8th International Conference on Learning Repre- sentations (ICLR), 2020. htt..httss:://penneett wttpre/iwweeeeeeeeet/ee//e////- httt://frummmiddooom f6RFDR
ew net
Nadia Burkart and Marco F Huber. A Survey on the Explainability of Supervised Machine Learning. Artificial Intelligence Reseurch, 70:245-317, 2021. doi: 10.aa13/jair... 122288. URL https:
7/dl. acm. wwt.o/////oooooooooooo htt///oi11111111 161./161331110 /atae/jjai..o. 12228
Diogo V Carvalho, Eduardo M Pereira, and Jaime S Cardoso. Machine Learning Interpretability: A Survey on Methods and Metrics. Electronics, 8(8), 832, 2019. ISSN 2079 9292, doi: 10.3390/ electronics0080832. URL CRLLhttps:://et /www mdpi com 2079 9292 8/8/832
Nelson Elhage Catherine Olsson, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Gan- guli, Zac Setfield-Dodde Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context Learning and Induction Heads. Trans- former Circuits Thread, 2022. and
former--cfoooerrr wwwprrmerrrrrcttttto pub/20227 ni--conexxt-eerree hnw.:/mmnnnnan--nnnnnnnnooo----- www.dutttnneeeeeeeeeeeeaeaeaaeee wwtp//iidxxxec. html
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What Does BERT Look At? An Analysis of BERT's Attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 276-286, 2019. doi: 10.18653/v1/
W19-4828. URL https: /www. ac web.
9-282
10

---

Published as a conference paper at ICLR 2024
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge Neurons in Pretrained Transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 8493 8502, 2022. doi: 10. 10.18663/11/20222.cc-oooo
/tttpps://aaaaee anthh ogy org. 2022. .all:222ccllll -log..8.8
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference
of the North American Chapter of the Association for Computational Linguisties: Human Lan-
guage Technologies (NAACL-HLT), pp. 4171-4186, 2019. doi: 10. 10.1866//w1/11-1442 URL
https: www. awwweb..w........... org httpp//w..ororaaaaaaoooogg////--
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid- linear units for neural network function approximation in reinforcement learning. Neural Networks, 107:3-11, 2018. ISSN
0893-6080. doi: 10.
URL https: /tpp:://www... www. scwencedireecc..oo.o
Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Sheer El Showk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Hernandez, Amanda Askell, Kamal Ndousse, Andy Jones, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, Zac Hatfield-Dodds, Jackson Kernion, Tom Conerly, Shauna Kravec, Stanislav Fort, Saurav Kadavath, Josh Jacobson, Eli Tran-Johnson, Jared Kaplan, Jack Clark, Tom Brown, Sam McCandlish, Dario Amodei, and Christopher Olah. Softmax Linear Units. Transformer Circuits Thread, 2022a. URL https://tzaasforrrss wwteemmee-rrrrure circuitscs..... pub/ html
/ 2022/ http///ool//nnnne/e//e
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfiel--Dodss, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy Models of Super- position. Transformer Circuits Thread, 2022b. URL
ci ciruits..
mub//2222ttyymmode/////
html
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Haffield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Ka- mal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread, 2022c 2tahb//222///e/ http://rawwwwrrarommooooooooooee index. html
URL https: http:::://taaassonnnoe wwraer-oorrrrrooo www.eeerrrrrrrtstee.ee
Javier Ferrando, Gerard I. Gállego, and Marta R. Costa-jussà. Mcasuring the Mixing of Contextual Information in the Transformer. In Proceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pp. 8698 8714, 2022. doi: 10. (0.18653/v12005 emnlp-main.598 mane-mainn555. 595.
www.thoooy.org//00.... org/2022 wwnle-malaam
595
Javier Ferrando, Gerard I. Gállego, Ioannis Tsiamas, and Marta R. Costa-jussà. Explaining How Transformers Use Context Build Predictions. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (ACL), pp. 5486-5513, 2023. doi: 10. 1/228333 URC/ittps://accaatttlloyy www.aanttttooooooooooooooooooeee acooon....lll...n..o...o. 301
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer
Layers Are Key-Value Memories. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 448--5455 2021. doi: 10. 10.18653/11/2211mmmmmpp 446.
URL https: http::::/aclaantoooog..ooooo 2021. wwnnp-manaa..... 446 http:///ww....roe222..oo...aaaaa
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual asso-
ciations in auto-regressive language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 12216 12235, 2023. doi: 10.18653/v1/ 2023.emnll-mainn7511 URL htRL:t/ts:///at antho orthhl0ly.org//20233 emm.0:2m...mmmm.mmmmmmmm. p- i 751
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A. Roberts. https:
The Unreasonable Ineffectiveness of the Deeper Layers. arXiv preprint, 2024. URL
//arxiv.oree warp:www.rg/abb/2203.111
11

---

Published as a conference paper at ICLR 2024
Dan Hendryeks and Kevin Gimpel. Gaussian Error Linear Units (GELUs). arXiv preprint, c5.LG//1606.08415544, 2016. doi: 10.485500axXi..1666.088111 URL https:/ bttEps://axxi..cct arxivv.rq////// abs/1606. 08415v5
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Purameter-ffficient Transfer Learning for NI.P. In Proceedings of the 36th International Conference on Machine Learning (ICML), press// PMIR,, volume 97, pp. 2790-2799, 2019. URL https: htttps://proceedinii
v97 297/houlsby11 19a 9tmttml.
Albert Q- Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie- Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B. arXiv preprint, 2023. URL
https://arxxt wttp:///////a/ooeee 06825v1
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gi- anna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of Experts. arXiv preprint, 2024. URL http::tt///// arxiv. org abs//41bb/4 04C88v1
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. Attention Not Only a Weight: Analyzing Transformers with Vector Norms. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pp. 7057-7075, 2020. doi: 11.18653/11/2222emmmpp 2020. e::.0mmmmppmmaa 574
BtLhhtps::/ /www//wwwwww.....//////ww www.wb..ww.....c................ wtw.eeb.ogggaaahoogggg.e
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. Incorporating Residual and Normalization Layers into Analysis of Masked Language Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4547- 4568, 2021. doi: 10. 10.18653/v1/202 10.16666///2222mmmmmmmmm....
2021. 1omnlp-main.377
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz- Richardson. Captum: A unified and generic model interpretability library for Py Torch. arXiv preprint, LLG/2009.07896 2020. (RLtps:::///aas org/ abs/ 07896v1
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky, Revealing the Dark Secrets of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPP IJCNLP), pp. 4364 4373, 2019, doi: 10. 18553//1/DD19114455 URL www./wwwaa.. web. hrrg/anthology////
1445
Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. BERT Busters: Outlier Dimensions that Disrupt Transformers. In Findings of the Association for Computational Linguis- tics: ACL-IJCNLP 2021, pp. 3392-3405, 2021. doi: 102188653///22221 11 findiings-acl.300, URL
https://aclanthon logy wtw.//2.....iiinnn.g.......... acl 300
2021.
Nikolaus Kriegeskorte, Marieke Mur, and Peter Bandettini. Representational similarity analysis connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2:4, 2008
doi: 10.3389/neuro. http://ewurr.000004..000000 06.004.2008.
URL https::/ //t::://www....
10. 3389/ neuro. 06. 004 2008
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy. Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pre- training Approach. arXiv preprint, es CL/CC/99...11112/1 2019. doi: 10.48550Vvriin,1107..116 URL http:://aara..oorg////////0.00//
12

---

Published as a conference paper at ICLR 2024
Shapley Lloyd S. A Value for n-person Games. Contributions the Theory of Games (AM-28), Vol- ume 11. pp. 307-318 1953. doi:
WRLLhttps:/ /www. jstor.
Scott M Lundberg and Su-In Lee. A Unified Approach to Interpreting Model Predictions.
In Advances in
Neural Information Processing Systems 31 ccppsp/pprppccccpppppppppe
(NewrIPS), pp. 4765-4774, 2017.
URL
7062 a unifunifie http:///mm....apoooooooeeeeeeeee
Ziyang Luo, Artur Kulmizev, and Xiaoxi Mao. Positional Artefacts Propagate Through Masked Language Model Embeddings. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11hh International Joint Conference on Natural Language Processing (ACL-IJCNLP), pp. 5312 5327, 2021. doi: 10. 00.18633/112021..cllbb
URL
413
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and Editing Factual Associations in GPT. In Advances in Neural Information Processing Systems 36 (NeurIPS), 2022.
URL http:tts///peereeeeee netreeiieeeeeeeeeeeeeeeee net forum?id Satprummidd---s--eeee--
Paul Michel, Omer Levy, and Graham Neubig. Are Sixteen Heads Really Bet- In Advances in Neural Information Processing Systems 33
ter than One? (NeurIPS), pp. 14014-14024 2019. URL https://papers nips. ct..ps.cccppprr////pp/ppp 9551 9551-are-2 si xww.e-eeneeeeee heads real 1y better-
Ali Modarressi, Mohsen Fayyaz, Yadollah Yaghoobzadeh, and Mohammad Taher Pilehvar. GlobEnc: Quantifying Global Token Attribution by Incorporating the Whole Encoder Layer in Transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACI-HILT), pp. 271-271, 2022. doi: 10. 10.18653//1120222aacc-miial
URL htttp:://cclatthol
org/2022 naacl naacl main.19
Ali Modarressi, Mohsen Fayyaz, Ehsan Aghazadeh, Yadollah Yaghoobzadeh, and Moham- mad Taher Pilehvar. DecompX: Explaining Transformers Decisions by Propagating Token De- composition. In Proceedings of the 61st Annual Meeting of the Association for Computa- lional Linguistics (ACL), pp. 2649-2664, 2023. doi: 1/22663//1/2222.alllleo 149. URL https: / ogg.orgg........ooo..o..o.......
wwgy.org/000. 2twrg/2222.c.om acl-1e.......nn........m 149
Vinod Nair and Geoffrey E. Hinton. Rectified Linear Units Improve Restricted Boltzmann Ma- chines. In Proceedings of the 27th International Conference on Machine Learning (ICML), pp. 807-814, 2010. URL https:: : / ////mmmm. wtt.eee/oooeeeeeee///00////////
pdd.
Daisuke Oba, Naoki Yoshinaga, and Masashi Toyoda Exploratory Model Analysis Using Data-
Driven Neuron Representations. In Proceedings of the Fourth BlackboxNLP Workshop on Ana- lyzing and Interpreting Neural Networks for NLP, pp. 518-528, 2021. doi: 10.18653/w1/0011
blackboxnlp- 1.41.
Byung-Dot Doh Oh and William Schuler. Entropy- and Distance- Based Predictors From GPT-2 Atten- tion Patterns Predict Reading Times Over and Above GPT-2 Surprisal. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9324 9334, 2022. doi: 10.18653/v1/2022 www.:22.emllpaaee wwnlp-main.633 URL https:/ http:://alaaanthloy.ooooo
2022. emmlp main. 632
Chris Olah. Mechanistic Interpretability, Variables, and the Importance of Interpretable Bases. Essay
in Transformer Circuits Thread, 2022. URL httph::tt///// trans woreer
Sub/2022/mechh interp essay
Giovanni Puccetti, Anna Rogers, Aleksandr Drozd, and Felice Dell' Orletta. Outlier Dimensions that
Disrupt Transformers are Driven by Frequency. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 1286- 1304, 2022. doi: 10.18653/v 0008655/112222/ 1/2022. condings-emnlp...
URL httpss//aac antho ogy. ogy.org/2022 f find wanndings--en ngs- 93
13

---

Published as a conference paper at ICLR 2024
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. In Proceedings of the 5th Workshop on Energy Efficient
Machine Learning and Cognitive Computing, 2020. URL https:/ /www:///wwwww. emcl- -anc2-ai.. org/
asset.s docs wweddd/sssuurre neur 1awpurrippp11/m22--2mm11111 9/emm22nuuu ps pappp-pppa.. 33.
Michael Santacroce, Zixin Wen, Yelong Shen, and Yuanzhi Li. What Matters In The Structured Pruning of Generative Language Models? arXiv preprint, 2023,
0as/2200.03377
Thibault Sellam, Steve Yadlowsky, Ian Tenney, Jason Wei, Naomi Saphra, Alexander D' A'mour, Tal Linzen, Jasmijn Bastings. Iulia Raluca Turc, Jacob Eisenstein, Dipanjan Das, and Ellie The MultBERTs: BERT Reproductions for Robustness Analysis. In 10th International
Pavlick.
Conference on Learning Representations (ICLR), 2022. URL Forum?id-KOE F00
Pratyusha Sharma, Jordan T. Ash, and Dipendra Misra. The Truth is in There: Improving Reasoning in Language Models with Leler-Selective Rank Reduction. arXiv preprint, 2023. URL https:
13558v1
Noam Shazeer. GLU Variants Improve Transformer. arXiv preprint, cs. cs.LLG/2002.05022 2020. URL
https://aaxiis www.//ggsss..ooooo 2202. 05202v1
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. Recursive Deep Models for Semantic Compositionality Over Sentiment Tree- bank. In Proceedings of the 2013 Conference on Empirical Methwds in Natural Language Pro- cessing (EMNLP), pp. 1631-1642. 2013, URL https::///// htwp:/wwwwwww.b.../.owwwwwpw. aclweb. org/anthology/ 913-1174
Erik Strumbelj and Igor Kononenko. An Efficient Explanation of Individual Classifications using Game Theory. Journal of Machine Learning Research (JMLR), 11(1):1-18, 2010. URL http::
7/jmlr. wttpp//pppp////////eeessp.
htmm
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic Attribution for Deep Networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), PMLR, volume 70, pp. 3319 -3288, 2017. URL
sundarara an html
Chaofan Tao, Lu Hou, Haoli Bai, Jiansheng Wei, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. Structured Pruning for Efficient Generative Pre-trained Language Models. In Findings of the
Association for Computational Linguistics: ACL 2023, pp. 10880-10895, 2023. doi: 10.18653/ wtw.:0223...nniing-cca-ca..
112003.findings. 6c2.
692
lan Tenney, Dipanjan Das, and Ellie Pavlick. BERT Rediscovers the Classical NLP Pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), org/
pp.
2019. doi: 10. 10.18653/v1/P19- 1452. URL htttps://ww..
anthology/PI9- 14527
Tulia Ture, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Well-Read Students I Learn Better: On the Importance of Pre-training Compact Models, arXiv preprint, es. CI1908 08962v2, 2019. doi: 10.48550/arXiv. 190...ii.110..... 1908.08962 URL pettts::/aarxet
org/ abs 908 08962v2
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Infor- mation Processing Systems 30 (NIPS), pp. 5998-6008, 2017. URL http:/paaprs.ipp.......
-you-needd
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-aa art natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP): System Demonstrations, pp. 38-45, 2020. URL https:// acl anthooooyy
www.hhlllooorgg//222222 wwt.//gg//22... emn 1-ddemos
14

---

Published as a conference paper at ICLR 2024
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On Layer Normalization in the Transformer Architec- ture. In Proceedings of the 37th International Conference Machine Learning (ICML), PMLR, volume 119, pp. 10524-10533, 2020. URL xiong20b.ht.r html
ngg.mng... mlr www.s..resssllll
Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A Survey on Knowledge Distillation of Large Language Models. arXiv preprint, 2024. URL httpt:tts::/aatt
org/ /20// 13116v3
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: Open Pre-trained Transformer Language Models. arXiv preprint, W.CCL/2205.010665 2022. URLltttps. /tatps://arxiii http://awwwww..gaabbba// 2205. 0106801066
A RELATED WORKS
This section supplements the studies directly related the methods or experimental results of this paper, which are not fully discussed in the main body.
A.1 MECHANISTIC INTERPRETABILITY FOR TRANSFORMER LMs
Mechanistic interpretability seeks to reverse engineer neural networks, similar to how one might reverse engineer a compiled binary computer program. (Olah) 2022
In the past decade, as deep learning has become the predominant approach for classification and rep- resentation learning, interpreting and understanding has emerged as a significant challenge for the NLP community. The NLP and closely related ma- chine learning communities have amassed a vast array of scientific and engineering insights into the interpretation of deep learning models under the banners of interpretability. explainability, XAI, and probing. Meanwhile, in diverging line of research, researchers from organizations such as Google, Anthropic. and OpenAl have published compelling research under the banner of mechanistic inter- pretability, focusing on the interpretation of internal parameters of deep learning models. Although these two approaches share many technical commonalities, the exchange between them is not yet as active as could be. In this section, we will overview the trend of mechanistic interpretability for the NLP/ML community and specifically highlight research that is closely related to this paper.
dineneinnniinnl,, nonlinear language models
Mechanistic interpretability begins with the analysis of toy or small-scale language models, aiming for thorough and comprehensive understanding of internal phenomena and mechanisms, interpret- ing parameters and activations, and providing tools for the analysis. The interest in mechanistic interpretability for Transformer language models (LMs) has been growing, particularly with An- thropic's s Transformer Circuits Thread Our study, the analysis of attention maps (mixing between tokens), focuses on how information from specific tokens propagates to surrounding tokens in the model. Therefore, our study may be seen as operations similar to the assembler's 's MOV and ADD commands from the perspective of mechanistic interpretability.
Additionally, the attention map is used as an analytical tool in mechanistic interpretability. Elhage al. (2022c) discovered specific attention heads (called Induction heads) through the observation of attention maps calculated with norm- norm-based analysis. Catherine Olsson et al. (2022) revealed that the induction heads contribute the in-context learning of large-scale language models.
In mechanistic interpretability, several studies have attempt to interpret FF (MLP) layers, but chal- lenges understanding and interpretation have been raised (Elhage et al. 2022b a). We believe that
our method or results for understanding FF through the lens of attention maps can be beneficial to the field of mechanistic interpretability.
httpsss/ttransffmrr--iirroo
15

---

Published as a conference paper at ICLR 2024
A.2 ATTENTION MAP (ANALYSIS OF MIXING BETWEEN TOKENS)
The core component of the Transformer is the self-Attention mechanism, and focusing on the inter- actions between tokens (attention maps) has become one of the mainstream analytical approaches. The computation of attention maps starts with attention weights and has been improved or ex- tended (Clark et al., [2199] Kovaleva et al. 2019: Brunner al., 2020 Kobayashi al. 2020 (021).
One of the key extension strategies is the norm-based analysis. which can consider the effects of all surrounding matrices, vectors, and modules without approximation, and also satisfies the axiom of the additive feature attribution methods (Lundberg & Lee, 2017). Initially, Kobayashi et al. (2020) proposed a norm-based computation method that considers the entire attention block, ATTN; subsequently, Kobayashi et al. (2021) expanded this approach to include the effects of the RESI 1 and LNI. For technical details, please refer to $ 2 The norm-based analysis provides insights into the internal mechanisms of models; for example, attention maps computed with norm- corm-based analysis used to measure similarities between language models and human behavior, e.g., reading time (Oh & Schuler 2022).
This paper proposes and analyzes a method for calculating attention maps across an entire layer. However, as related research future work, it is also possible calculate attention maps across the entire Transformer architecture. |Abnrr & Zuidema (2020) computed an attention map at each layer by adding identity matrix to the attention weights matrix (accounting for ATTN and RES1) I), and then integrated these maps (all the layers) in two ways: (i) Attention rollout uses recursive matrix multiplication and (ii) Attention flow uses a maximum flow algorithm. |eerando et al. (2022) proposed ALTI, which uses norm- banedd analysis to compute attention maps at each layer considering the Attention block (ATTN, RES1, I,NI), and integrates them using Attention rollout. Modarressi et al. (2022) proposed GlobEnc, which uses horm-based analysis to compute Attention maps at each layer considering all modules except FF (i.c., ATTN, RESI, LNI, RES2, and LN2), and integrates them using Attention rollout. Modarressi et al. (202) proposed DecompX, which decomposes the processes of the entire model similarly to the norm- based analysis and calculates an attention map for the entire model relative the model's prediction (logit).
DecompX (Modarressi et al. 2023) is the only method among those previously mentioned that considers the nonlinear activation functions in the FF layers. Therefore, from this perspective, De- compX can be considered a competitor to our proposed approach. We will discuss the differences in greater detail in the following. DecompX uses coarse linear approximation for decomposing the non-linear function in FF. Our study differs from DecompX in two aspects: (i) the scope of the study (visualization tool for model-wide behavior vs. analysis of module function) and (ii) the de- composition of FF (with vs. without approximation). Investigating the impact of the approximation in the decomposition of FF on our analysis an important future work.
A.3 OUTLIERS IN TRANSFORMERS
$ 6.2 demonstrated that FF and LN2 exhibit unique behaviors concerning outliers. is well-known that the internal representations Transformer LMs tend to have outliers (Luo et al. 2021 Kovaleva
et al. 2021). It has also been revealed that FF generates outliers in certain dimensions (Ferrando
jet al. 2023
Bondarenko al. 2023), and that the parameters of LN contains outliers (Modarressi Puccetti et al. 2022). However, to our knowledge, our finding that FF and L.2 cancel
et al.
2022
out each other through these outliers is novel and suggests significant insights into the redundancy of the model.
A.4 REDUNDANCY OF TRANSFORMERS
As mentioned above, our results ($ 6.2 suggest redundancy in the Transformer architecture. Redun- dancy of Transformer has also been uncovered from the success of parameter reduction. Successful pruning has been achieved for the ATTN (Michel et al. 2019 Kovaleva et al. 2019), FF (Santacroce et al. 2023), and both of them (Gromov al. 2024 Tao al. 2023). Additionally, attempts have been made to reduce learned parameters through low-rank approximations (Sharma et al.) 2023 and
to construct lightweight models using knowledge distillation (Sanh et al. 2020 al. 2024
16

---

Published as a conference paper at ICLR 2024
B CONCRETE EXAMPLES OF OUR ANALYSIS
Figs. 9 and 10 show concrete examples of the attention maps calculated using our analysis method for BERT and GPT-2. Fig. shows that FF in the 12th layer of BERT amplifies the mixing from "Tokyo" to " [MASK]" for the input " [CLS] Tokyo is the capital of [MASK] [SEP]". Fig. IO shows that FF in the 3rd layer of GPT-2 amplifies the mixing from "rather" to "than" for the input "She loves math rather than engaging in".
00.4
12
(21)
(BP))
(BP)
1000
ATB
FFF
+ RES
+FF++ +RES+IN
Figure Attention maps computed with our analysis method for 12th layer BERT-base.
+LN+FF
+LX+FF+RE
Figure 10: Attention maps computed with our analysis method for 3rd layer in GPT-2 small.
C DETAILED FORMULAS FOR EACH ANALYSIS METHOD
We describe the mathematical details of the norm-based analysis methods adopted in this paper.
C.1
ATB
As described in $ Kobayashi et al. (2021) rewrite the operation of the attention block (ATB) into the sum of transformed inputs and a bias term:
y ATB (TB X)
(15)
LN1 RES1 ATTN
(16)
F ATB (zy: X) + ATB
(17)
First, the multi-head attention mechanism (ATTN: Eq. 1 can be decomposed into sum of trans-
formed vectors as Eq. B|and Eq. 7
ATTN(2; D X)
ΣΣ (2, + bb Wo
bo + bo
(18)
Σ FATTN (my; X) + ATTN
(19)
EATTN X)= = W
(20)
ATTN by Wo + bo.
(21)
17

---

Published as a conference paper at ICLR 2024
Second, the residual connection (RES; Eq. 4 just performs the vector addition. So, ATTN and RES also can be decomposed into a sum of transtormed vectors collectively:
(RESI o (TT X) (TTN(E;; X) + æ;
(22)
Σ
FATTN (m); X) + 2;; + BATTN
(23)
j7i
AATNNREES
(zy; X) + AATTN
(24)
FATTN 23j X) FATTN 7 X + 32
( (=
FATTN+RES (z):
(25)
Third, the layer normalization (LN; Eq. 5 performs a normalization and element- affine trans-
formation. Suppose the input to LN sum of vectors = Σ; Zy. LN's operation decomposed as follows:
2 m(z)
IN(2)=
7 G
(26)
8(2)
= 7 -
k= ((k)
① 0
(27)
s(z)
Σ, 2j à k=1
(E, 21) (1)
B
(28)
8(2)
Σ 2j 1 E (k)
07+B
(29)
5(2)
Σ 2j m(z))
07+3
(30)
s(z)
Σ FLN(24 (zz) +B
(31)
žj ml (2))
FLN (zj)
7
(32)
(z)
where (A) denotes the k-th element of vector z. By exploiting this decomposition, attention block
(ATB; ATTN, RES, and LN) can be decomposed into a sum of transformer vectors:
(LN1 RES1 ATTN)(#;; X) LN
F ATTN RES
X) + BATIN)
(33)
Σ FLN ATTN+REE (z): + FLN (BATTN)
+ B (34)
Σ PATB (mys X) + ATT
(35)
FATB (xy; X) = FLN (F. FATTNNRES az;; X) b ATB = FUN (BATTN) +33.
(36)
(37)
Then, the ATB method quantifies how much the input æ, impacted the computation of the output ATB by the norm FATB (z,; X)I
C.2 ATBFF
The feed-forward network (FF) via a two-layered fully connected network (Eq, 3). By exploiting
the decomposition of an activation function g ($ 3 ATB and FF can be decomposed into sum of
18

---

Published as a conference paper at ICLR 2024
transformed vectors collectively:
(FF ATB) (z; X) g
ATTB D1 W)W + b1 W2 + b2
(38)
11
9
Fj
FATB (zy; X) + b W1 + b,
W2 + b2
(39)
F FABB (z): X)W, +
+b ATBB W1 + b1
W2 + b2
(40)
n
g
F.te Pre 9 (2x)) ) + bree W2 b
(41)
hj
F. Pro ( (x1)) (..,,..., F. (ze)): b 9 g.00 0
F.
+ h1
F. Baaai ( ... F. 1 (xx)) gre 9; (..
W2 + b2 (42)
hj
Pre (
(x,)) bro 9: 0. W2
(43)
j=1
+ h4
F (I))) F. F (x,,, b g: 0 W2 b2
(44)
F ATB+FF (xz; X) + FAT++FF
(45)
F.th (wy; X) F:ABB(a (zy; X)WI
(46)
greg ATT W, bi
(47)
FATB-FF (org; X) hn (2)1) F (xr,); Bre "g W₂
(48)
6ATB+FF- hbs P. a (xi),.... E. (æn), 6 pre 0 W2 + 62.
(49)
Then, the ATBFF method quantifies how much the input æj impacted the computation of the output y, ATR-FF by the norm F ATB+FFF (z;; Detailed decomposition of the activation function g is described in Appendix D
C.3 ATBFFRES
The residual connection (RES; Eq. 4 just performs the vector addition. So, ATB, FF, and RES2 also can be decomposed into a sum of transformed vectors collectively:
(RES2 FF ATB) (m;; X) (FF ATB) 0y; X) + ATB (x;; X)
(50)
Σ F ATB+FF
(x;; X) + AAB (rg X) + ATB+FF
(51)
J
Σ
F ATB+FF+ RES
(æy; X) FTB+FF
(52)
F ATB+FFF (m;; X)
(+
FAT (x
F. FFF
(a; X) + ATB X)
(53)
Then, the ATBFFRES method quantifies how much the input æ; impacted the computation of the
output y, ATB++F++ES by the norm FTT+++FF+ (my; X)]-
19

---

Published as a conference paper at ICLR 2024
C.4 ATBFFRESLN
By exploiting the decomposition of LN (Eq. 31 and Eq. (2), entire Transformer layer (ATTN, RES1,
LNI, FE, RES2, and LN2) be decomposed into sum of transformer vectors:
Layer( [e;; X) IN
F PATB+FF+RES
(By; X) BATB+FF)
(54)
Σ FLN PAIB+EPTRES (et;; x)) + FLN (bATBAFF) +
(55)
Σ Flayer (ay;; X) + b Layer
(56)
Fl Flayor (x.z X) FLN F. FAIB+EFARES (az x))
(57)
Layer
FLN (BATB-FF) + B.
(58)
Then, the ATBFFRESLN method quantifies how much the input my impacted the computation of the layer output y: by the norm
F. playyer (xzj; X)Il-
D DETAILS OF DECOMPOSITION IN S 3
D.1 FEATURE ATTRIBUTION METHODS
One of the major ways to interpret black-box deep learning models is measuring how much each input feature contributes to the output. Given a model f: Rn -> R;; a: r> f(xx) and certain input
a' = ( r,,), this approach decomposes the output f(xx) into the sum of the contributions q; corresponding to each input feature z/:
f(x') = ei +... cr(xf
(59)
Interpretation methods with this approach are called feature attribution methods (Carvalho et al, 2019 Burkart & Huber, 2021), and typical methods include Integrated Gradients (Sundararajan et al. 2017) and Shapley value (Lloyd S. 1953). Comparison between Integrated Gradients and Shapley value described in Appendix
D.2 INTEGRATED GRADIENTS (IG)
IG is an excellent feature attribution method in that it has several desirable properties such as com- pleteness 3). Specifically, IG calculates the contribution of each input feature x by attributing the output at the input æ' I2 relative to baseline input b E R":
0
(G.(.'ff.b) = (< be)
da.
(60)
a Ox:
b+ afw -b)
This contribution IG,(æ'; f.b) satisfies equation 59
f(x') = f(b) + +Σ (G,
(61)
The first term /(b) can be eliminated by selecting baseline b for which f(b) - (see D.3)
D.3 DECOMPOSITION OF GELU WITH IG
This paper aims to expand the bosm- analysis (Kobayashi et al. 2020), the interpretation method for Transformer, into the entire Transformer layer. However, simple approach decom- pose the network in closed form as in Kobayashi et al. (2020) (021) cannot incorporate the activation function (GELU, Hendrycks Gimpel 2016) contained in FF This paper solves this problem by decomposing GELU with IG (Sundararajan al. 2017).
Many recent Transformers, including BERT and RoBERTa, employ GELU their activation function.
20

---

Published as a conference paper at ICLR 2024
GELU: R - R is defined as follows:
I ( +
VT |.-
GELU(x) = 2
e
c dt
(62)
Considering that in the Transformer layer, the input a E of GELU can be decomposed into sum of zj terms that rely on each token representation of layer inputs (x: + ... Fn), GELU can be viewed as multivariable function GELU:
GELU (21............... ................................ Fn) ) = G zz).
(63)
Given a certain imput x' x + ry, the contribution of each input feature x' to the output GELU(x') is calculated by decomposing GELU(Z),..... with IG (Eq. 60 and 610
GELU(x') GELUGE (4
-ド IG, (2 (ff oiis GELU, 0),
(64)
where b = 0 was chosen. Note that GELU(() = and the last term in equation 61 is eliminated.
practical neural network implementation, the GELU layer is passed a vector æ1 + +.................. 2n R instead of a scalar 21 +... + In and the GELU function is applied (broadcasted) element-wise. Let GELU: Rdd Rd be defined as the function that broadcasts the clement-level activation GELU: R- + R. The contribution of each input vector z/ = [2') ..................... a') [d']] to the output vector GELU + ... + xf,)) is as follows:
Decomposition of broadcasted GELU
GELU(* [1] + +... s [1])
GELU(x + +... x²,)
(65)
GELU(a [1') +... x.,
Σ
GELU, 0)
(66)
Ej= (G [d GELU, 0)
(G x [1]: GELU, 0)
Σ
(67)
_nn [d'l GELU, 0)
The above decomposition is applicable to any activation function that passes through the origin and differentiable in practice, and covers all activation functions currently employed in Transformers,
such as ReLU (Nair & Hinton 2010). SiLU (Elfwing et al. 2018), and SwiGLU (Shazeer) 2020).
E COMPARISON BETWEEN FEATURE ATTRIBUTION METHODS
We decomposed the nonlinear activation function using Integrated Gradients (IG), a feature attribu- tion method (see $ 3 and Appendix D) which is motivated by its desirable property completeness to be combined with norm-based analysis (& 33. Nevertheless, there is another typical attribution method, Shapley Value (Lloyd S. 1953) (SV), that satisfies this property as well. In this section, we compare IG with SV in the context of our norm- norm-based analysis.
E. SHAPLEY VALUE
Shapley Value (SV) was originally introduced in cooperative game theory. and this concept has been imported into the machine learning field to compute specific input attribution. In specific setting (Eq. (63). the sum elements = Ij + ... + In is fed into a nonlinear function (c.g., GELU). Each element x, [k] can be considered as player in cooperative game, and SV can
21

---

Published as a conference paper at ICLR 2024
calculates its contribution to output. A well-known XAI method based on SV is SHapley Additive exPlanation (SHAP. Lundberg & Lee 2017). which extends SV to apply to neural models. In our setting, SV and SHAP are equivalent (strictly speaking, SV is equivalent to SHAP with zero inputs as a reference input) because the target process can be viewed as cooperative game as well as a neural model (since it starts with sum of inputs).
E.2 IMPLEMENTATION OF SV AND IG
SV: SV computation in Python can be achieved in several ways using the shap library (Lundberg
& Lee 2017 We used three SV methods: (i) Exact, (ii) Sampling, and (iii Kernel methods. (i)
Exact method is an exact computation of the Shapley value, computing the output of a nonlinear function for all combinations (O(2") of input elements (Ir (Ir [k., F.n F..[.))))))))) (R:) to compute the expected
change in output with without each element. (ii) Sampling method (Strumbelj & Kononenko 2010) approximates the computation with only the sampled combinations instead of all combina- tions. shap library samples 2 n + 2048 combinations by default. (iii) Kernel method (Lundberg & Lee (017) formulates the SV calculation as a regression problem and approximates its calculation by sampling. It also samples 2 2048 combinations by default.
IG: Our IG computation was achieved in Python using Captum library (Kokhlikyan et al., 2020 11 In particular, the Gauss-Legendre quadrature method was used as the method of integral calculation (selectable argument).
E.3
COMPARISON BETWEEN IG AND SV
We use IG and SV (three methods above) with our methods and compare the obtained performance and results. We experimented on NVIDIA GeForce RTX 2080 Ti and Intel Xeon Silver 41 12.
First, we compared their speed (execution time) using BERT-tiny and three inputs of length n = 5, 10, 15. Any of the three SV methods required more time for longer input; specifically, these required more than 15 minutes to process the input of length n = 15, which is a typical (or shorter) length of subword-segmented single sentence. On the other hand, IG required less than 1 second for any input of them (see Appendix F for detailed calculation cost). Hence, SV requires huge execution time for longer inputs and larger models, then SV could not be employed in this study.
Next, we compared their yielding results using BERT-tnny and 10 input of length < 15 sampled from the validation set of SST-2 dataset. Fig. IT shows results of macro comtextualizatton change by FF, RES2, and LN2 ($ 5.I for IG and SV. We can see that IG and SV (three methods) yielded similar results. This result suggests that the results and claims in this paper are not be significantly affected by the choice of attribution methods.
1:00
1.00
BERT-tiny
Change
0.75
0.75
0.75
0.50
0.50
Kamal
0.50
Kems!
0.25
0.00
Avg.
Avg.
2 Layer
Avg
Layer
Layer
(a) +FF
(b) +RES
(c) +LN
(ATB 4+> ATBFF)
(ATBFF < ATBFFRES)
(ATBFFRES < ATBFFRESLN)
Figure 11: Contextualization changes between before and after each component in FFBs (FF,
RES2, and LN2) of BERT-tiny calclulated with IG and SV (Exact, Sampling, and Kernel methods). The higher the bar, the more drastically the token-to-tokes contextualization (attention maps) changes due the target component.
16 hhttesss/4tthhbb.co/////////aaa
22

---

Published as a conference paper at ICLR 2024
F COMPUTATIONAL COST OF OUR ANALYSIS
We test the computational costs of our method against inputs of various lengths in six BERT variants of different sizes. We fed each sample of the Wikipedia dataset into each BERT model and measured the time to our analysis. We experimented on NVIDIA GeForce RTX 2080 Ti and Intel Xeon Silver 122. Fig. [2 that the cost increase (running time) against input length becomes sharper when using larger models (hidden dimension and number of layers). That is, the cost increases interactively with the length of the input and the size of the model. This means that our method would be costly to apply to billion-scale LLMs, and lightening the computation an important future work.
16-
tiny
mini
small
Time [sec]
medium
10-
base
8-
large
W% as Toai Bout
01
40
80
100
120
Input length
Figure 12: Running times of our analysis method for six BERT variants of different sizes and inputs of various lengths.
G SUPPLEMENTAL RESULTS CONTEXTUALIZATION CHANGE
We reported the contextualization change through each component in FFBs of BERT-base and GPT- 2 on the Wikipedia dataset in $ [5] We will report the results of the other models/datasets in this section. In addition, we provide full results of linguistic patterns FF's contextualization effects that could not be included in the main body.
G.1 MACRO CONTEXTUALIZATION CHANGE
The contextualization changes through each component in FFBs of six variants of BERT models with different sizes shown in Fig. [4] The results for four variants of BERT-base models trained with different seeds are shown in Fig.
The results for two variants of RoBERTa models with different sizes shown in Fig. T6 The results for OPT 125M model are shown in Fig- 17 The results for BERT-base and 2 on the SST-2 dataset are shown in Fig. [8] These different settings with other models/datasets also yield similar results reported in 5.1 each component did modify the input contextualization, especially in particular layers. The mask language models showed consistent trend of larger changes by FF and LN in the middle late layers. On the other hand, the causal language models showed a trend of larger changes by FF in the early layers.
G.2 LINGUISTIC PATTERNS IN FF'S CONTEXTUALIZATTOO EFFECTS
Tables and are the extended versions of Table T showing the top ten word-word pairs with the highest FF-amp scores in each layer of GPT-2 and BERT.
Fig. 13 shows the distribution word-word pair types (e.g., composing the same word) in the top 50 token pairs with the highest FF-amp score, which is measured on the SST-2 dataset. While the overall trend similar to the results on Wikipedia dataset (Fig. 5). there are some minor differences (e.g., subword and same token categories). These differences
be attributed to differences in the datasets. The Wikipedia dataset is composed of excerpts from Wikipedia articles, while the SST-2 dataset is composed of movie reviews. Experimenting with other datasets and models is future work.
23

---

Published as a conference paper at ICLR 2024
subword
frequent expression
VIA
named entity
others
compound noun
same token
N00
semantical connection
100
100
% pairs
% pairs
8
10 12
FR
5
8
30
11
12
PR
AR
BERT Layer
GPT-2 Layer
Figure 13: Breakdown of the category labels we manually assigned to top 50 pairs having the largest FF-amp score each layer of BERT and GPT-2 with SST-2 dataset. We also assigned the labels to fully random 50 pairs ("FR") and adjacent random 50 pairs ("AR").
H SUPPLEMENTAL RESULTS DYNAMICS CONTEXTUALIZATION CHANGE
We reported the dynamics of contextualization change BERT-base and GPT-2 on the Wikipedia dataset in 6 We will report the results of the other models/datasets in this section.
H.1 CONTEXTUALIZATION CHANGE THROUGH FF AND SUBSEQUENT COMPONENTS
The contextualization changes through by FF and subsequent components, RES and LN, in six variants of BERT models with different sizes are shown in Fig. 19 The results for four variants of BERT-base models trained with different seeds are shown in Fig.
20 The results for two variants The results for OPT 125M model are shown in Fig. 22 The results for BERT-base and GPT-2 on the SST-2 dataset are shown in Fig. 23 These different settings with other models/datasets also yield similar results reported $ through the FF and subsequent RES and LN, the contextualization change score once becomes large but finally converges to small; that is, the contextualization by FFs tends to canceled by the following components.
of RoBERTa models with different sizes are shown in Fig. 2T
H.2 FF AND RES
The L2 norm of the output vectors from FF and the vectors bypassed via RES2, in six variants of BERT models with different sizes are shown in Fig. 24 The results for four variants of BERT-base models trained with different seeds are shown in Fig.
25 The results for two variants of RoBERTa The results for BERT-base and GPT-2 on the SST-2 dataset are shown in Fig. 27 These different settings with other models/datasets also yield similar results reported in 6.11 the vectors bypassed via RES2 are more than twice as large as output vectors from FF in the L2 norm in more than half of the layers. That is, the representation (contextualization) updated by the FF tends to be overwritten/canceled by the original one.
models with different sizes are shown in Fig. 26
H.3 FF AND LN
Mean absolute value in each dimension of the input/output vectors of FF and the weight parameter of LN at the layer where FF's contextualization effects are strongly cancelled by LN, in six variants of BERT models with different sizes are shown in Fig.
28
The results for four variants of BERT-base
models trained with different seeds are shown in Fig.
29 The results for two variants of RoBERTa
models with different sizes are shown in Fig. 30 The results for BERT-base and GPT-2 on the SST-2 dataset are shown in Fig. 31 These different settings with other models/datasets also yield similar results reported in $ 6..2 the FF output representation has outliers in some specific dimensions (green lines in the figures), and the weight of LN tends shrink these special dimensions (red lines in the figures). In the layers where FF incurs a relatively large impact on contextualization, the Pearson correlation coefficient between LN' r and mean absolute value of FF output by dimension was from -0. 0..88 to -0.93 in BERT-large, from - 0..2 to -0.47 in BERT-medium, from -0.56 to -0.76 in BERT-small, from -0.55 -0.73 in BERT-mini, -0.71 in BERT-tiny, from - 0.69 to -0.74
24

---

Published as a conference paper at ICLR 2024
in BERT- base (seed 0), from -0.48 -0.74 in BERT-base (seed 10), from -0.50 to -0.68 in BERT-base (seed 20). In these layers of RoBERTa models, the Pearson's 's r was small: from -0.02 to -.228 in RoBERTa-large and from 0.01 to -0.14 in RoBERT-base. However, the Spearman's 's p was large: from -0.46 to -0.56 in RoBERTa-large and from 0.80 to -0.94 in RoBERT-base.
In 6.2 also observed that ignoring such special outlier dimensions (bottom 1% with the low- est value of 7) in calculating FF s contextualization makes the change score quite small. The contextualization changes by FF when ignoring the dimensions are shown in Fig. 32
Table 2: Word pairs for which FF amplified the interaction the most in BERT. The text colors aligned with word pair categories: subword, compound noun, common expression, same token, named entity, and others.
Layer
Top amplified token pairs
(#ifouu, det), (fifiquue, ##munn, (#ficend, trans), (outer, space), (#filli, res), (#ilific, honor), (#Hnate, #tfimi), (opera, soap), (deco, art), (#iniggh, week)
(#ftroy, con), (guard, national), (america, latin), (casily, could), (#foy,, f), (marshall, islands), (#ffite, rec), (channel, english), (finance, finance). (ffffert, rev)
3 .D., D. (##mel, ##ons), (hut, ##chin), (water, ##man), (paso, ##k), (avoid, ##ant), (toys, #thop),, (competitive, ##ness), (##la, ##p), (#ittree, ##t)
(#B1, ds), (HI1, #ltera), (#fczz #flavee, (HHr, ##lla), (#ffle, beth), (#Heent, #Hili), (#Her, #ffbur), (#Hett, mn), (#Hiee, wai), (#Hrana, ke)
(##entt ##ili), (#Hious, #*car), (#Hon, ##ath), (##), ds), (#Hr,, ##ense), (#ffence, #Wili), (res, #Hili), (#ttrative, #Hjo)), (##ci,, ##pres), (##able, ##vor)
(on, clay), (with, charged), (#Hci, ##lpres), (be, considers), (on, behalf), (fleming, colin), (#Hh,, #illea), (-. clock). (#Htur, nam), (by, denoted)
7 (##onn, ##mmel), (vi, saint), (vi, st), (#H1), #Wifc), (##ti,, jon),
(#fii, wii), (Hiten, #ffchel), (##son, bis), (vessels, among), (#Wheer #ffrn)
(#Hfano, #fflz), (bo, #tilz). (nathan, or). (arabia. against), (sant, fffini). (previous, unlike), (saudi, against), (#ffia, #ffuringg, (he, gave), (tnt, equivalent)
(no, situation), (##tk,, ezech), (according, situation), (decided, year), (v, classification), (eventually, year). (but, difficulty), (#ffference, track), (she, teacher), (might, concerned)
10 (jan, ##nen), (hee, ## 1), (primary, stellar), (crete, crete), (nuclear, 1991), (inspector, police), (##tu, ##nen), (quote, quote), (f, stellar), (v, stellar)
11 (tiny, tiny). (#ffwater, ##water), (hem, hem), (suddenly, suddenly), (fine, singer), (henley, henley), (highway, highway), (moving, moving), (dug, dug). (farmers, agricultural)
12 (luis, luis), (tong, [l), (board, judicial, (#Hi index), (--). (four, fifty), (cloud, thunder), (located, transmitter), (##ota, ##o), (#Hss, analysis)
25

---

Published as a conference paper at ICLR 2024
Table 3: Word pairs for which FF amplified the interaction the most in GPT-2. The text colors are aligned with word pair categories: subword, compound noun, common expression, same token, named entity, and others.
Layer
Top amplified token-pairs
(ics, stud), (ning, begin), (ever, how), (ents, stud), (ited, rec), (une, D). (al, sever), (itions, cond), (ang. p), (ree, c)
2 (al, sever). (itions, cond), (z. jan). (ning, begin), (-was, this), (er, care), (ies, stud), (une, j), 1990), (ents, stud)
3 (.than, rather), (z. jan), (_then, since), (bin, ro), (long, how), (ling, rever), (une, j), others. among). (arks, cl), ( into. turning)
(.same, -multiple), (ouri, _iiss), (-jung, lex), (.b, sttrngghh), (Lothers, .among), (.bssed, ""), (adesh, hra), (-top, among). (t, give), (bin, ro)
5 (ol. brist), (ol. ink), (gal, man), (ac, pens), (op, link), (.whit, &), (ant, avvidd), (people, .important.. (un, _bra), (,1995, 1194)
(ol, ink), (-del, del), (o, .dec), (ist, oh), (it, me),
rouue, routc), (thel, in), (ord, m), (adier, division), (we, ob)
7 las), (che, rr), (43, aGi). (green, .mark), (.cer, family), (jack, eter), (orses, (Jiilometres, _lies,, G. _disappointed)), (umer, in)
(ar, goddess), d, ight), 33, density). cer, family), 08, 2007),
(bid, _family), (_14, _density), (.c, _faiily, (_50, -yield), (_15, .density)
9 (.ray, ffrnnch,, (_11, uly), (Lone, _score), (on, _french), (_41, (_37,-%), (.equipped, _another), (bc, -i). (ad, en), (.,. te)
10 (ho, ind), (.by, ashi), (_bra, _von), (Land, _317), (ad, en), (Lloves, ashi), (Lu, asaki), (_he, ner), (Lland, ines), (-de, die)
(it, "). (uman, (whenn. (ave, _daughters), (.bird, .:), (ic, there), (ia, _counties), (bc, -:), (Land, had), (.2010, _us), (T.:)
(ooperational, _not), (Lanswered, .therefore), (-degrees. having), ( (Lvarying, having), (k, rant), ( supplied, also), daring, has), prominent, 's), (a. ually), ( stress. :)
26

---

Published as a conference paper at ICLR 2024
1.00
1.00
1.00
BERT- Larr-
Change
0.75-
0.50
0.50
2525
0.25
0.25
0.00
1
M
0.00
0.00
Avg.
Avg
Avg
Layer
Layer
Layer
(a) +FF
(b) +RES
(c) +LN
(ATB ATBFF)
(ATBFF + ATBFFRES)
(ATBFFRES ATBFFRESLN)
1.00
BERT-base
Change
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
101112 Avg
10:112
Layer
Layer
Layer
(d) +FF
(c) +RES
(f) +LN
(ATB ++ ATBFF)
(ATBFF ++ ATBEFRES)
(ATRFFRES + ATRFFRESLN)
BERT-mediom
1.00
1.00
1.00
Change
0.75
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
Avg.
Avg.
Avg
Layer
Layer
Layer
(g) +FF
(h) +RES
(i) +LN
(ATB ATBFF)
(ATBFF ATBFFRES)
(ATBFFRES + ATBFFRESLN)
1.00
1.00
1.00
BERT-smal
Change
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0
0.00
Aug.
Avg
Layer
Layer
Layer
(j) +FF
(k) +RES
()) +LN
(ATB +> ATBFF)
(ATBFF +> ATBEFRES)
(ATRFFRES + ATRFFRESIN)
1.00
1.00
1000
BERI-mini
Change
0.75
0.75
0.75
0.50
0.50
0.50
025-
0.00
0.00
0.00
Avg.
2
Avg.
2
3
Avg.
Layer
Layer
Layer
(m) +FF
(n) +RES
(0) +LN
(ATB 4+ ATBFF)
(ATBFF ATBFFRES)
(ATBFFRES + ATBFFRESLN)
1.00
1.00
1.00
BERT-tiny
Change
0.25
0.25
0.25
0.00
0.00
0.00
Avg.
Avg.
Layer
Layer
Layer
(p) +FF
(q) +RES
(r) +LN (ATBFFRES ATBFFRESLN)
(ATB ATBFF)
(ATBFF + ATBFFRES)
Figure 14: Contextualization changes between before and after each component in FFBs (FF, RES2, and LN2) of six variants of BERT models with different sizes (large, base, medium, small, mini, and tiny). The higher the bar, the more drastically the token-to-token contextualization (attention maps) changes due the target component.
27

---

Published as a conference paper at ICLR 2024
1.00
1:00
1.00
Original
Change
0.50
0.50
0.25
0.25
0.00
0.00
0.00
101112ANg
101112 Avg.
Layer
Layer
Layer
(a) +FF
(b) +RES
(c) +LN
(ATB ATBFF)
(ATBFF +> ATBFERES)
(ATBFFRES ATBFERESLN)
1.00
1.00
Change
0.75
0.75
0.75
Seed
0.50
0.50
0.50
0.25
0.25
0.25
0.00
201112
101112
101:12
Layer
Layer
Layer
(d) +FF
(c) +RES
(f) +LN
(ATB 4> ATBFF)
(ATBFF 4> ATBFFRES)
(ATBFFRES 4 ATBFFRESLN)
1.00
1.00
1.00
Seed I0
Change
0.75
0.75
05
0.50
0.50
0:50
0.00
0.00
3
101112 Avg
2 3
5
101112
2
101112 Avg
Layer
Layer
Layer
(g) +FF
(h) +RES
(i) +LN
(ATB +9 ATBFF)
(ATBFF ATBFFRES)
(ATBFFRES + ATBFFRESLN)
1.00
1.00
1.00
Change
0.75
11
Seed
0.50
0.50
0
0.25
0.25
0:5
0.00
0.00
0.00
Avg
Layer
Layer
Layer
(i) +FF
(k) +RES
(I) +LN (ATBFFRES + ATBFFRESLN)
(ATB ATBFF)
(ATBFF ATBFFRES)
Figure 15: Contextualization changes between before and after each component in FFBs (FF, RES2, and LN2) of four variants of BERT-base models trained with different seeds (original, 0, 10. and 20). The higher the bar, the more drastically the token-to-token contextualization (attention maps) changes due to the target component.
28

---

Published as a conference paper at ICLR 2024
RoBERTa-land La-laaee
1.00
1.00
1.00
Change
0.75
0.75
0.75
0.25
11
0.00
0.00
-Bq---h
10
15
20
Avg.
10
Avg
10
15
20
Avg.
Layer
Layer
Layer
+FF
(b) +RES
(c) +LN
(ATB ATBFF)
(ATBFF ATBFFRES)
(ATBFFRES ATRFFRESLN)
1.00
1.00
1,00
RoBERTTa-base
Change
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
Avg.
Avg.
Layer
Layer
Layer
(d) +FF
(c) +RES
(i) +LN (ATBFFRES + ATBFFRESLN)
(ATB 4>> ATBFF)
(ATBFF ATBFFRES)
Figure 16: Contextualization changes between before and after each component in FFBs (FF, RES2, and LN2) of two variants of RoBERTa models with different sizes (large and base). The higher the bar, the more drastically the token-to-token contextualization (attention maps) changes due to the target component.
1.00
1.00
1.00
OPT 125M
Change
0.75
0.75
0.75
0.50
0.50
0.50
0.00
0.00
0.00
2
101112 Avg
101112 Avg.
101112 Avg.
Layer
Layer
Layer
(a) +LN
(b) +FF
(c) +RES (ATBLNFF ATBLNFFRES)
(ATB <> ATBLN)
(ATBLN <> ATBLNFF)
Figure 17: Contextualization changes between before and after each component in FFBs (FF, RES2, and LN2) of OPT 125M model. The higher the bar. the more drastically the token-to-token contextualization (attention maps) changes due to the target component.
29

---

Published as a conference paper at ICLR 2024
BERTa-base (Post-l LN)
1.00
1:00
Change
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0:25
1 2
6
101112
12 3
101112
201112
Layer
Layer
Layer
(a) +FF
(b) +RES
+LN (ATBFFRES t> ATBFFRESLN)
(ATB 4> ATBFF)
(ATBFF t> ATBFFRES)
1.00
1.00
1.00
GPT-22 (Pre-LN)
Change
0.75
0.75
0.75
0.50
0.00
0.00
2 4
7 101112Avg
1
101132AN9,
2
6 201112 Avg.
Layer
Layer
Layer
(d) +LN (ATB ATBPRELN)
+FF (ATBLN ATBLNFF)
+RES (ATBLNFF ATBLNFFRES)
Figure 18: Contextualization changes between before and after each component in FFBs (FF, RES2, and LN2) of BERT-base and GPT-2 on SST-2 dataset. The higher the bar, the more drastically the token-to-token contextualization (attention maps) changes due to the target component.
30

---

Published as a conference paper at ICLR 2024
+ FF
+Fr
+FF RES
+FF+
+FP 1FP++RES RES +LN (Same layer's Postl
+ FF++ RES + I.N (Same layer's Postl.N)
1.00
1.00
0.75
0.75
Change
Change
0.50
0.50
0.25
0.25
0.00
10
15
20
Avg.
1
3
5 7
9 1011 Avg.
Layer
Layer
(a) BERT-large.
(b) BERT-base.
+FF
+FF
+FF RRS
RES
+FP RES
+FF RES +LN (Same layer's PostLN)
+FF + RES + LN (Same layer PostLN)
1.00
1.00
0.75
0.75
Change
Change
0.50
0.50
0.25
0.25
0.00
0.00
5 Layer
Avg.
Avg.
Layer
(c) BERT-medium.
(d) BERT-small.
+FF
+FF
4F++RRS
RES
+FF RES (Same layer's Postl N)
FF RES I.N (Same layer's Postl N))
I.N
1.00
1.00
Change
0.75
Change
0.75
0.50
0.50
0.25
0.25
0.00
0.00
2
3
Avg.
Avg.
Layer
Layer
(c) BERT-mini.
(f) BERT-tiny.
Figure 19: Contextualization changes through processing each component (FF, RES, and LN) from just before FF (ATB) six variants of BERT models with different sizes (large, base, medium, small, mini, and tiny). The higher the bar, the more the contextualization (attention map) changes from just before FF.
31

---

Published as a conference paper at ICLR 2024
+FF
+FF
+RE
+ +FF +ES
RES LN layer's
+ +FF +RS +INN (Same layer's PostLN)
1.00
1.00
Change
0.75
Change
0.75
0.50
0.50
0.25
0.25
0.00
11 10
0.00
1011 12 Avg.
6
10 Avg.
Layer
Layer
(a) BERTa- base (original).
(b) BERTa- base (seed 0).
FF
+ FF
+FF + RES
+ +FP RES
RES + I.N layer's Postl.
+ +F++RE + +IN layer's PostLN)
1.00
0.75
Change
Change
0.50
0.50
0.25
0.25
InDe 0
0.00
0.00
2
4
5
6
7
8
9
10
11
12 Avg.
1
2
4
6
8
10
10111. 12 Avg.
Layer
Layer
(c) BERTa-base (scod
BERTa-ban base (seed 20).
Figure 20: Contextualization changes through processing each component (FF, RES, and LN) from just before FF (ATB) four variants of BERT-base models trained with different seeds (original, 0. 10, and 20). The higher the bar, the more the contextualization (attention map) changes from frottjust before FF.
+ FF
+FF
+
#FF++RE
+ FF RES
+RE
layer's
+ FFr +RES + LN (Same layer's PostL.N
1.00
1.00
Change
0.75
Change
0.75
0.50
0.50
0.25
0.25
10
0.00
0.00
10
20
Avg.
6 Layer
10 Avg.
Layer
(a) RoBERTa-large.
(b) RoBERTa
Figure 21: Contextualization changes through processing each component (FF, RES, and LN) from just before FF (ATB) in two variants of RoBERTa models with different sizes (large and base). The higher the bar, the more the contextualization (attention map) changes from just before FF
32

---

Published as a conference paper at ICLR 2024
+ FF
FF +RE
FF RES LN (Next layer's PreLN)
1.00
Change
0.75
0.50
0.25
0.00
1
2
3
4
6 9 10 12 Avg.
Layer
Figure 22: Contextualization changes through processing each component (FF, RES, and LN) from just before FF (ATBLN) in OPT 125M model. The higher the bar, the more the contextualization (attention map) changes from just before FF.
+FF
+FF
+FF+RES
FF RES
+FF+ RES + LN (Same layer's PostI.N)
+FF RES LN (Next layer's PreLN)
1.00
1.00
0.75
0.75
Change
Change
0.50
0.50
0.25
0.25
0.00
2
10 Avg.
6
8 Layer
10 12 Avg.
Layer
(a) BERT-base.
(b) GPT-2.
Figure 23: Contextualization changes through processing each component (FF, RES, and LN) from just before FF (ATB) in BERT-base and GPT-2 SST-2 dataset. The higher the bar, the more the contextualization (attention map) changes from just before FF.
33

---

Published as a conference paper at ICLR 2024
300
60
RE52
RES2
Norm
Norm
100
20
Avg.
Avg.
Layer
Layer
(a) BERT-large
(b) BERT-basc.
FF
RES2
RES2
20
Norm
Norm
15
10-
Awg
Layer
Layer
(c) BERT-medium.
(d) BERT-small.
FF
15
FF RES2
Norm
Norm
10
10
5
Layer
Layer
(c) BERT-mini.
(f) BERT-tiny.
Figure 24: Averaged norm of the output vectors from FF and the bypassed vectors via RES2, cal- culated on the Wikipedia data for six variants of BERT models with different sizes (large, base, medium, small, mini, and tiny).
RES2
60
RES2
Norm
Norm
40
20
10
12
Avg
Layer
Layer
(a) BERT-base (original).
(b) BERT-hase (seed 0).
FF
FF
RES2
Norm
Norm
Layer
Layer
(c) BERT base (seed 10).
(d) BERT- base (seed 20).
Figure 25: Averaged norm of the output vectors from FF and the bypassed vectors via RES2, cal- culated the Wikipedia data for four variants of BERT-base models trained with different seeds (original, 0, 10, and 20).
34

---

Published as a conference paper at ICLR 2024
50
40
RES2
RES2
Norm
Norm
30
20
Agg.
11
Avg.
Layer
Layer
(a) RoBERTa-large
(b) RoBERTa-base.
Figure 26: Averaged norm of the output vectors from FF and the bypassed vectors via RES2, cal- culated on the Wikipedia data for two variants of RoBERTa models with different sizes (large and base).
RES2
RES2
Norm
Norm
Aug
Avg.
Layer
Layer
RoBERTa-base
(b) GPT-2.
Figure 27: Averaged norm of the output vectors from FF and the bypassed vectors via RES2, caleu- lated on the SST-2 dataset for BERT-base and GPT-2.
35

---

Published as a conference paper at ICLR 2024
2.0
Mean absolute value
30
2.0
output weight
wel welght ght value
weight
10
1.0
-10-
-20
0.5
-20
-30
200
400
600
800
1000
0.0
200
400
600
800
Dimension
Dimension
(a) 18th layer in BERT-large.
(b) 10th layer in BERT-base.
FF input FF cutbut wiight
Mean absolute vo ue
FT Inpur
weight value
FF output
LN weight value
1.5
10
NN weight
1.0
Mean
-20
0.0
-20
0.0
Dimension
Dimension
(c) 5th layer in BERT-medium.
(d) 3rd layer in BERT-small.
absolute value
2.0
Mean absolute value
10
outpuc
LN weight value value
weght
1.5
1.0
0.5
input
0.5
-10
ourput
N weight
-15
50
300
150
200
250
0.0
-10
0
20
40
60
80
100
120
0.0
Dimension
Dimension
(c) 3rd layer BERT-mini.
(f) 1st layer in BERT-tiny.
Figure 28: Mean absolute value in each dimension of the input/output vectors of FF across the Wikipedia data and the IN weight values the certain layer of six variants of BERT models with different sizes (large, base, medium, small, mini, and tiny).
FF irput
Mean abso ute value
curput
LN welght nt value
1.0
1.0
-10
-10
FF input
200
100
Dimension
Dimension
(a) 10th layer in BERT-base (original).
(b) 9th layer in BERT-base (seeed 0).
Mean absolute value
imput
2.0
Mean absolute value
Inpul
2.0
FF output
value
40
output LN weight
weight value
10
LN weight
1.5
100
40
200
400
600
600
0.
-60
200
400
600
800
0
Dimension
Dimension
(c) 10th layer BERT-base (seed 10).
(d) 11th layer in BERT-base (seed 20).
Figure 29: Mean absolute value in each dimension of the input/output vectors of FF across the Wikipedia data and the LN weight values at the certain layer of four variants of BERT-base models trained with different seeds (original, 0, 10, and 20).
36

---

Published as a conference paper at ICLR 2024
FF input
Mean absolute value
40-
FF Input
FF outpu
FF outputt
LN weight va ue
155
Mean absolute
10
LN weight
1.0
-10-
-40
Dimension
Dimension
(a) 1st layer in RoBERTa-large.
(b) 10th layer in RoBERTa-base.
Figure 30: Mean absolute value in each dimension of the input/output vectors of FF across the Wikipedia data and the LN weight values at the certain layer of two variants of RoBERTa with different sizes (large and base).
input FF ourput unight
Mean absolute value
inpul FF output
2.0
value
30
LN weight value ue
15
NN
Mean absolute
10
-20
200
400
800
-15
400
800
Dimension
Dimension
(a) 10th layer in BERT-basc.
(b) 4th layer GPT-2.
Figure 31: Mean absolute value in each dimension of the input/output vectors of FF across the SST-2 dataset and the LN weight values at the certain layer.
1.00
1.00
Change
0.75
Change
0.75
0.25
0.25
0.00
0.00
3
4
7 Layer
101112
2
4
6
101112 Avg.
Layer
(a) ATB ATBFF in BERT-basc.
(b) ATBFF ATBFFRES in GPT-2
Figure 32: Contextualization changes by FF of BERT-base and GPT-2 on Wikipedia dataset when the 1% (seven) dimensions with the smallest LN weights 7 values is ignored the norm calcula- tion. The higher the bar, the more drastically the token-to-token contextualization (attention maps) changes due to the target component.
37


Reviews:
[{"review_id": "BSnyN78Hon", "reviewer": "Reviewer_Ur6S", "summary": "The paper proposed methods to analyzed the feed-forward blocks with regards to the input contextualization. It leverages the completeness property of existing norm-based analysis and the integrated gradient method.\n\nThe motivation to analyze the feed-forward blocks include the following:\n1. The feed-forward blocks account for 2/3 of the layer parameters.\n2. There is a growing interest in feed-forward blocks (new approaches focusing on the feed-forward blocks)\n3. Previous work reported that feed-forward blocks perform some linguistic operations\n\nTheir experiments using masked-LM and casual-LM have shown that feed-forward blocks modify the input contextualization by amplifying specific types of linguistic compositions. Feed-forward blocks and layer normalization largely control contextualization. \n\nThey also found that feed-forward block and other blocks cancel out each other's contextualization effects, which might indicates redundancy in the Transformer computations. (Feed-forward blocks' effects are weekend by surrounding residual and normalization layers.)", "questions": "I think it might help the reader with a description or definition of contextualization.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is the first to analyze the whole feed-forward blocks, including the non-linear activation function. The non-linear activation function has been previously excluded from the norm-based analyses because it cannot be decomposed additively by the distributive law.\n\nCombining the norm-based and the integrated gradients, the paper is able to quantify and visualize the effects of the whole feed-forward block.\n\nThe paper also provides detailed analysis that points to interesting properties of the feed-forward blocks. The discovery of redundancy might leed to new improvement of the architecture.", "weaknesses": "The author pointed out that the future work might be working with the latest large language model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dFCxnY3xfx", "reviewer": "Reviewer_Dtpy", "summary": "This paper proposes a way of analyzing feedforward blocks (=FFB) [including residual connections and the layer norm layers intervening] extrapolating from the typical attention maps of the mid and late 2010s and the slightly more recent concept of a \"refined\" attention map that also considers the values and output projection (Equation 8). They call this base formulation ATB. The authors further extend this notion to incorporate FFBs. The FFB's nonlinearity which makes it additively non-decomposable [a-la linearity of expectation] is overcome by using the integration gradients paper [Sundararajan et al, 2017]. The three resultant formulations, which incorporate just the linear, thenceforth residual connections and layer norm are christened ATBFF ATBFFRES and ATBFFRESLN. The Pre-Layer-Norm variants are [like the post-LayerNorm one before] named likewise. \n\nThe authors analyze a good variety of encoder only LM checkpoints in addition to a decoder only lM checkpoint [GPT2-117M] , which is also an instance of the pre-LN formulation.\n\nThe authors then interestingly proceed to analyze the contextualization changes caused by the FF block , both in terms of extent of change [based on flattening the pairwise values and taking Spearman Correlation of before-after], linguistic contextualization, as well as the dynamics.", "questions": "- What is the computational [and memory] complexity of generating these maps? This may sound nitpicky but with increasingly large LLMs which barely fit in the accelerator time and memory bounds whether at training or inference time, this can indeed become a factor and consideration in how widely this gets adapted.\n- I know mechanistic explanation is a somewhat orthogonal paradigm of interpreting large transformer architectures, but it would be nice to have some comments on how this can relate or synergize with that paradigm [if at all]\n- What is the effect of banded local attention [alternating banded local sparse and global attention] are a common part of the architecural recipe in many GPT3 or later LLMs so this would be a valuable insight to have.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Their formalism is extended to both pre and post Layer Norm variants of transformers.\n- Tested on a large variety of encoder LM architectures.\n- Decoder-only architectures are also covered [though just one, i.e. GPT2-117M].", "weaknesses": "- It would have been nice if the authors could have discussed and potentially also experimented with atleast one alternative formulation to IG, or atleast one of its variants [They do mention other formulations in B.1 Appendix but did not see further broaching of this angle beyond this]\n- [Doesn't apply after authors comprehensively addressed this on 20th Nov] A marginal weakness but one nonetheless [and this is alluded to in future work], would have been nice to see this for a new-age LLM, of which some variants are available at lower or comparable parametrizations to GPT2-117M (e.g. OPT-125M)\n\nPost-Script: The authors exhaustively addressed the second point! Thanks for that!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lMLHkrRpJT", "reviewer": "Reviewer_wwew", "summary": "This paper proposes to use attention map to analyze the effects of feed-forward blocks in transformers. Different from previous works which mostly focus on studying attention weights, the authors leverage norm-based analysis and integrated gradient methods on the the effect of feed-forward networks, residual connection, and layer normalization. Experiments done one different models (including different sizes of BERT and RoBERTa) and on different dataset (Wikipedia and Stanford Sentiment Treebank) suggest that feedforward networks (FF) amplify specific types of linguistic compositions, and its surrounding components tend to cancel out each other's contextualization effects.", "questions": "1. Why is b set to a zero vector in equation 12?\n2. Why do you think the patterns of changes between BERT and GPT-2 are quick different in each component (e.g., from Fig. 3)? What does this entail for different architectures?\n3. Are results from Section 5 averaged across positions? Would position representation bias your findings?\n4. How is the micro contextualization change (by subtracting a pre-FF attention map) in Section 5.2 different from measuring correlations?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper studies the contextualization effects of feed-forward blocks by leveraging attention maps, which were mostly ignored before (by only looking at attention weights). This presents new views on analyzing what each block in transformer is functioning.\n2. The findings that FF and surrounding components tend to cancel out each other's effects are interesting. This may provide more perspectives in designing and training transformer models.", "weaknesses": "1. The paper suggests that because of the cancelled out effects in surrounding components there is \"potential redundancy in transformer layers\". However, there is not enough evidence to justify this claim (e.g., by training a transformer model removing some of the components). Without more experiments and results, it is not convincing what the conclusion of this paper is. More importantly, there is no systematic evaluation on the linguistic patterns (e.g., linguistic patterns distribution from different datasets on each layer) apart from some sampled amplified pairs. The results presented in Table 1 and Figure 5 are not evident.\n2. The results on different BERT sizes, and different seeds do not seem to be always consistent (e.g., Figure 9, 10 in the appendix).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1qY73zHh8d", "reviewer": "Reviewer_GB37", "summary": "This paper focuses on analyzing Feed-Forward (FF) blocks in the Transformer model, specifically regarding their impact on input contextualization. The authors utilize a refined attention map by combining norm-based analysis and the integrated gradient method, which offers completeness in understanding the FF block's behavior. In the experiment, Wikipedia excerpts and Stanford Sentiment Treebank v2 datasets are used, and the authors analyzed 11 11 masked LMs and one casual LM. The results show that FF blocks do modify input contextualization by amplifying specific linguistic compositions, such as subword pairs forming a single word. Furthermore, the authors discover that the FF block and its surrounding components tend to cancel out each other's contextualization effects, shedding light on the mechanism and suggesting redundancy in processing within the Transformer layer.", "questions": "What could be the challenges of applying the proposed method in larger models, such as OPT, LLaMA, as mentioned in the future work?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Analyzing Feed-Forward (FF) blocks in Transformer models through a refined attention map is novel, which combines norm-based analysis and the integrated gradient method. While previous research has explored the behavior of FF blocks, this study offers a unique perspective by leveraging the aforementioned techniques to gain a comprehensive understanding of their impact on input contextualization. \n\n2. The experiments conducted with masked and causal language models demonstrate the effectiveness of the approach in capturing the modification of input contextualization by FF blocks. The clarity and precision of the analysis enhance the quality of the research, ensuring reliable and valid results.\n\n3. The paper is clear and easy to follow. The description of the refined attention map, norm-based analysis, and integrated gradient method is presented in a clear and understandable manner. The experiments and their results are well-explained, enabling readers to grasp the implications of FF block behavior on input contextualization.", "weaknesses": "1. To enhance the clarity of the proposed method, it would be beneficial to provide a running example that illustrates the step-by-step process. By walking readers through a concrete example, they can more easily grasp the methodology and its application.\n\n2. To better ground the paper in the existing literature, it would be valuable to provide a more detailed and comprehensive literature review. By thoroughly reviewing relevant prior research, the paper can establish its position within the broader academic discourse and highlight its unique contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)