A Lens into Interpretable Transformer Mistakes via Semantic Dependency
Authors: Ruo-Jing Dong
Yu Yao
Bo Han
Tongliang Liu
Venue: ICML 2025 poster (2025)
Keywords: token-level semantic depdendency
transformer
Primary area: social_aspects->accountability_transparency_and_interpretability

Abstract: Semantic Dependency refers to the relationship between words in a sentence where the meaning of one word depends on another, which is important for natural language understanding.
In this paper, we investigate the role of semantic dependencies in answering questions for transformer models, which is achieved by analyzing how token values shift in response to changes in semantics.
Through extensive experiments on models including the BERT series, GPT, and LLaMA, we uncover the following key findings:
1). Most tokens primarily retain their original semantic information even as they propagate through multiple layers.
2). Models can encode truthful semantic dependencies in tokens in the final layer.
3). Mistakes in model answers often stem from specific tokens encoded with incorrect semantic dependencies. Furthermore, we found that addressing the incorrectness by directly adjusting parameters is challenging because the same parameters can encode both correct and incorrect semantic dependencies depending on the context.
Our findings provide insights into the causes of incorrect information generation in transformers and help the future development of robust and reliable models.

A Lens into Interpretable Transformer Mistakes via Semantic Dependency
Ruo-Jing Dong'
Yu Yao
Bo Han
Tongliang Liu
Abstract
generate false information (Lin al., 2021), exhibit halluci- nations (Ji et al., 2023), leak sensitive training data (Carlini et al., 2021), and even engage in deception (OpenAI, 2023; Scheurer et al., 2024). Addressing these issues has led to the development of evaluation methods for L.LM performance (Liang et al., 2022) and strategies aimed at mitigating harm- ful outputs (Ganguli et al., 2022; Bai et al., 2022).
Semantic Dependency refers to the relationship between words in sentence where the meaning of one word depends on another, which is impor- tant for natural language understanding. In this paper, we investigate the role of semantic depen- dencies in answering questions for transformer models, which is achieved by analyzing how to- ken values shift in response changes in seman- tics. Through extensive experiments models including the BERT series, GPT, and LLaMA, we uncover the following key findings: 1). Most to- kens primarily retain their original semantic infor- mation even as they propagate through multiple layers. 2). Models can encode truthful semantic dependencies in tokens in the final layer. 3). Mis- takes in model answers often stem from specific tokens encoded with incorrect semantic depen- dencies. Furthermore, we found that addressing the incorrectness by directly adjusting parameters is challenging because the same parameters can encode both correct and incorrect semantic depen- dencies depending on the context. Our findings provide insights into the causes of incorrect infor- mation generation in transformers and help the future development of robust and reliable models.
Existing research has elucidated several reasons for the mistakes observed in LLMs. Studies have suggested that non-linearity, insufficient model averaging, and inadequate regularization lead to mistakes when encountering crafted adversarial examples (Chakraaorrty et al., 2018; Zhang et al., 2020). Additionally, the programmatic behavior of LLMs may lead to vulnerabilities under security attacks and pro- duce harmful content (Kang al., 2024). Competing objec- tives and mismatched generalization may cause the suscep- tibility of safety-traine LLMs (Wei et al., 2024). Studies also indicate various reasons for language models generating unfaithful or nonsensical text, including cource-reference divergence in data, imperfect representation learning, erro- neous decoding, exposure bias, and parametric knowledge bias (Ji et al., 2023). These studies have identified various reasons that lead to mistakes and enhanced our understand- ing, providing valuable insights into model weaknesses. Building upon these insights, we aim to further explore the internal mechanisms within the model's architecture that lead to mistakes.
1. Introduction
We believe that mistakes produced by LLMs arise from the way semantic dependencies, the relationships where the meaning of one token depends on another (Mel'čuk, 2001), are encoded within transformer models. Intuitively, in transformer models, inputs are tokenized and embedded as vectors that carry semantic information. These tokens then pass through multiple attention layers, where they ex- change and aggregate semantic information build seman- tic dependencies. These dependencies are crucial for the model's contextual understanding and reasoning, enabling it to generate coherent outputs. Since the model's predic- tions ultimately depend on the final layer outputs, which are constructed from the representations generated by earlier layers, any inaccuracies in the propagation and exchange of semantic information can result in misrepresented semantic dependencies. Such mistakes disrupt the model's ability to accurately understand token relationships and contextual
Large Language Models (LLMs) based on the trans- former architecture such as BERT (Devlin et al., 2018), GPT(Radford et al., 2019; Brown, 2020), and LLaMA (Tou- vron et al., 2023) have demonstrated remarkable capabilities across various natural language tasks. Alongside their bene- fits, L.LMs pose significant risks and challenges (Weidinger et al., 2021). For example, LLMs may intensify biases (Navigli et al., 2023; Taori & Hashimoto, 2023), produce toxic content (Gehman et al., 2020; Ousidhoum et al., 2021),
'Sydney AI Centre, The University of Sydney "TMLR Group, Hong Kong Baptist University. Correspondence to: Tongliang Liu <tongliang, liu @ sydney.cdu.aa
Proceedings of the 4 International Conference on Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A Lens into Interpretable Transformer Mistakes via Semantic Dependency
meaning, thereby leading to mistakes in predictions.
dependencies between tokens to understand sentence-level meaning. For example, the model must recognize how adjec- tives modify nouns or how subjects relate to actions. In the case of the imput "red apple and blue sky" shown in Figure 2, an output token will encode the semantically dependent information "red" and "apple" together, rather than encod- ing semantically independent information like "blue" and "apple". We find our evaluated models can encode truthful semantic dependency in the final layer tokens.
To systematically explore the role of semantic dependencies in causing model mistakes, we propose a method to inter- pret the semantic information aggregation mechanisms of transformer models. The idea is that altering the semantic information of an input token should result in significant changes the outputs of tokens that depend on this imput in- formation, while tokens that are not semantically dependent remain relatively unchanged. By evaluating the variations in output token representations in response to perturbations in imput tokens, we can effectively trace the flow and aggre- gation of semantic information throughout the model. This approach allows us to gain insights into how semantic de- pendencies are encoded and propagated within transformer models and helps identify potential disruptions that may contribute prediction mistakes.
rhinos
grey
instead
layer
color
rhinos
model answer: grey
layer L
grey
white
color
white
rhinos
grey
instead
white
layer
what
color
white
rhinos
Key Findings. In our exploration, we analyzed different transformer models such as BERT, LLaMA, and GPT. Here, we explain several key findings regarding the behavior of to- kens for semantic information aggregation and propagation.
color
rhinos
model answer: white
white
white
layer L
correct answer: grey
grey
color
Figure 3: An Illustration of Finding 3.
layer
the
token
aggregates
semantic
information
3). Finally, we found that when the model makes mistakes, certain tokens incorrectly encode information that is not semantically dependent. For example, Figure 3 demon- strates that semantic information is aggregated differently in the output token sequence when the model outputs an incorrect answer. In a question -atbeionnnn task where the context sequence "white rhinos are grey instead of white" is paired with the question "What is the color of white rhi- nos?", the correct answer is "grey". However, when the model incorrectly outputs "white", the question's 's key terms, such as "color" and "rhinos", contain more information about "white" rather than "grey". This highlights how false semantic dependency encoded in key tokens can lead to incorrect outputs.
aggregates
layer
token
semantic
Figure 1: An Illustration of Finding 1.
1). Most tokens primarily retain their original seman- tic information, even as they pass through the layers of transformers. For example, in Figure the arrows indicate the semantic information flow from the token at layer 0 to token at layer L. For the token "aggregates" in the input token sequence in layer 0, the final layer's token aggregates a large amount of information from its input token and a small amount of information from other tokens. The fact that most tokens still predominantly reflect their initial se- mantics highlights model's 's strong retention property, which is not inherently expected given the iterative aggregation of semantic information across many layers.
4). Additionally, we also find that the encoded semantic dependencies within a token are highly sensitive to both irrelevant context changes and the order of contexts. Due to space limitations, a detailed introduction, experiments, and results are provided in Appendix A.2.
layer
red
apple
blue
sky
Implications Our insights into semantic dependency en- coded within tokens of transformer models potentially help design new transformer architectures to be more resilient and semantically coherent. For example, our finding reveals that model mistakes often result from certain tokens erro- neously encoding semantic dependencies that should not exist. To address this, future research could refine attention mechanisms to better prioritize meaningful token interac- tions and reduce the impact of the adversarial context. This could be achieved by implementing dynamic re- weighting
red apple
apple
and
blue
sky
layer
sky
blue
Figure 2: An Illustration of Finding
2). A token in the final layer usually encodes truthful semantic dependency. Beyond preserving individual token semantics, the model must also encode truthful semantic
2

---

A Lens into Interpretable Transformer Mistakes via Semantic Dependency
strategies in top QA task-specific attention heads we lo- calized and incorporating stricter regularization techniques that can prevent tokens from erroneously encoding false dependencies while cautiously preserving the truthful ones.
retrieving factual information (Wu et al., 2024) or property- specific attention heads in CLIP models (Gandelsman et al., 2024). Our work offers another perspective by interpreting model mistakes via token-level semantic dependency en- coding, which provides insights into understanding and cor- recting model mistakes under specific suestion-ansserrig cases. Additionally, our finding in mutual attention heads responsible for key dependencies in QA tasks also shows the importance of pruning parameters in attention heads without tempering the correct semantic dependency encoding.
Our paper is structured as follows: ify a critical prerequisite for studying token-level semantic dependency: whether final-layer token retains its original semantic information. Because to define matched depen- dency between an input token (e.g., red) and another token (e.g., apple), (e.g..pplee) the corresponding final-layer token apple or red must reflect its original meaning. The result confirms our method's validity to capture semantic dependencies across multiple layers. Next, in Section 4, we systematically verify if models can encode truthful semantic dependency. Our finding indicates while models generally possess this foundational capability, mistakes still occur. Therefore, in Section 5, we further investigate how mistakes arise from false semantic dependencies in QA tasks and develop method to pinpoint a group of attention head parameters responsible for token-level semantic dependency. We tes- tify that addressing the incorrectness by directly pruning parameters is challenging because the same parameters can encode both correct and incorrect semantic dependencies.
first, in Section 3, we ver-
Probing Study for Linguistic Properties in Transformers. Probing methods (Rogers et al., 2021) analyze the internal representations of pre-trained rre-trained models to determine whether specific linguistic properties are encoded. For instance, re- search shows that BERT captures syntactic tree structures (Hewitt & Manning, 2019), semantic roles, and entity types (Tenney, 2019). Studies also quantify the mutual infor- mation between representations and linguistic properties (Pimentel et al., 2020). Token ablation is a parameter-free probing technique, which is widely used to analyze syntac- tic subtree structures (Wu al., 2020), syntactic agreement (Finlayson et al., 2021), and bias (Vig et al., 2020). Natu- rally occurring morpho-syntactic perturbations is also used to probe dependencies (Amini et al., 2023). Unlike these studies, which focus on static linguistic features, we investi- gate token-level semantic dependency encoding, introducing a framework to quantify dependency strength without prior knowledge. Our approach captures context-sensitive seman- tic dependencies, which can vary across diverse scenarios.
2. Related Work
Semantic Dependency Parsing and Semantic Role La- beling. Semantic dependency parsing (SDP) (Björkelund et al., 2010: Dozat & Manning. 2018) aims to identify se- mantic relationships between words in a sentence. Closely related to SDP, semantic role labeling (SRL) (He et al., 2017: Chen et al., 2025) focuses on identifying the predicate- argument structure of a sentence by assigning roles to words or phrases. However, the internal mechanisms by which transformer models encode, propagate, and utilize semantic dependencies remain largely opaque. Our work bridges this gap by exploring how internal mechanisms contribute to semantic dependency encoding and how these insights can be leveraged to address mistakes
Feature Attribution and Binding Study. Feature attri- bution methods primarily aim to assess the importance of individual tokens or features to the model's 's output. For example, prior work on attention flow (Abnar& & Zuidema, 2020) quantifies token importance through accumulated attention matrices. Gradient-based techniques like Conser- vative Propagation (Ali et al., 2022) is used to assess token attribution. Our study shifts focus from token importance to the semantic dependencies encoded in token representations and how these affect model behavior.
Semantic Information Flow in Transformer. Existing works (Liao et al., 2021; Schuster et al., 2022; Elhoushi et al., 2024) have studied activation stability and the limited con- tribution to token refinement in later layers of transformer models. However, whether the last-layer token retains its original semantic information in the input layer remains un- explored. Previous study (Geva et al., 2023) analyzes how factual associations are recalled while our study addresses a gap by studying how semantic dependencies are encoded in tokens and influence QA tasks.
Existing semantic dependency methods based on fea- ture/token interactions (Eberle et al., 2020; Janizek et al., 2021; Schnake et al., 2021) mainly focus on studying the contribution of combinations of features or tokens model predictions. Meanwhile, feature binding methods (Feng & Steinhardr, 2023; Vasileiou & Eberle, 2024; Wattenberg & Viégas, 2024) often do not test whether the model's most confident output reflects encoded semantic dependencies; rather, many assume this relationship holds and study down- stream properties. In contrast, our method is designed to explicitly test the assumption by evaluating whether there is dependence between the model's output and the semantic dependency encoded in the final-layer token.
Interpretable Model Mistake Based on Attention Heads. Previous studies highlight the roles of specific attention heads in model performance, such as retrieval heads for
3

---

A Lens into Interpretable Transformer Mistakes via Semantic Dependency
3. Most Tokens Primarily Retain Their Original Semantic Information Through Transformer Layers
Based on the above equation, we identify two underexplored mechanisms for validation. 1). Seff-information retention: validate whether the i-th token z in the output layer primar- ily retains information about the i-th token z in the input layer. Specifically, we compare the changes of all tokens in the final layer L with the changes in z If z changes most significantly when z changes, it suggests the i-th token in the final layer contains most information derived from the i-th token in the first layer. 2). Sequence-level semantic ag- gregation: validate whether token in L-th layer aggregates semantic information from tokens of the entire sequence =0. If every token change in 2n leads to the change of 2 it suggests z! contains information from all tokens.
In this section, we introduce a perturbation-bon based method to explore some mechanisms of semantic information propaga- tion in transformers. Extensive experiments show 1). even through multiple transformer layers, most final-layer tokens still primarily maintain their original semantic information from the first layer; 2). every final-layer token contains se- mantic information from almost all tokens (including itself) of the entire sequence. The results indicate our proposed method can effectively capture semantic changes in the in- put text and is suitable for detecting token-level semantic dependencies.
Token Perturbation We then generate K perturbed ver- sions of the input token z ( by only replacing the i-th token zς with randomly sampled tokens from the vocabulary V. Specifically, we sample a new token 2, follows.
Transformer Architecture We consider a general L-layer transformer model. Each layer consists of a multi-head self- attention mechanism (MHA) followed by position- wise feed-forward network (FFN), along with residual connec- tions. The imput sequence of N tokens is embedded into D-dimensional vectors and combined with positional en- codings to form the initial representations:
D(kk) for k times as
original z ( = [z]..... 2i. 2g1:
perturbed z0(k) = [Z]. 2.
((k
(4)
where z((A) Uniform(V) and E f1...., K}.
z = |z]. 2Q.................... ................................ 2.yl.
(1)
where z! E RDD is the embedding of the i-th token in 0-th layer.
Each perturbed sequence of token (0()) is processed inde- pendently through the I--laer transformer model, yielding L-layer token g((k), Similarly, the corresponding L-layer token for z ((rg) is z (org)
In transformer-based models, the token sequence is updated through L layers using the following two steps, where multi- head attention (MHA) and feed-forward networks (FFN)
work together to enrich the text representations:
Measuring Semantic Dependency To quantify how the perturbation of the i-th token z! in the first layer affects j-th token 2! in final layer, we examine the average change of the j-th token across the sequences. Specifically, for the j-th token, we calculate the semantic dependency score A Auflag which is achieved by calculating average change A Aal between its value in the original sequence and its values in the perturbed sequences:
z' = MHA (z' )
z' FFN' +2', (2)
where / = 2....... L. Here, MHA' and FFN' denote the meati-head attention and feed-forward network operations at layer L, respectively. The residual connections ensure that information flows directly through layers, facilitating the retention of original semantic information. For the i-th token in the output of the L-th layer, we have:
K
L
A 10/
K
((k) L(org) %
2
2
(5)
2 = -2.-
(2 + Σ (2)),
(3)
1=1
A higher value of 223 indicates that the j-th token in final layer is more sensitive to change of the i-th token. implies that j-th token encodes more information from the i-th token, i.c. encodes a stronger semantic dependency.
where !HA and FEN! represent the operations affecting the i-th token at | (Vaswani et al., 2017). Here we use the formulation proposed in the study (Gandelsman al., 2024), which ignores the layer-normalization term. The above equation shows that last-layer token can be written as a combination of first-layer tokens. This suggests that a last-layer token incorporates a varying yet unquantified amount of semantic information derived from the entire token sequence. Our research aims to address this gap by measuring such token-level semantic contribution.
To validate that the j-th token z/ in the output layer L encode strongest semantic dependency with the i-th to- ken in the input layer z,, we compare the average change A Aap for all tokens. If A o is the largest among all A 1 j E (1...., ., N), we can determine the j-th token in the final layer encode strongest semantic dependency with i-th token. For the first validation across multiple instances,
4

---

A Lens into Interpretable Transformer Mistakes via Semantic Dependency
Table I: Two validations on basic mechanisms of token- level semantic information propagation. Validation 1 is for self-information retention. Validation 2 for sequence-level semantic aggregation.
almost every final-layer token receives semantic information from every token (including itself) in the sequence.
4. A Final-layer Token Encodes Truthful Semantic Dependency
Model
validation 1 (%)
validation (%)
BERT (encoder
98.81
39.29
In the previous section, we observed that most tokens pri- marily retain their original semantic information. However, we also found that tokens not only retain their own semantic information but also integrate semantic information from all other tokens. In this section, we aim to verify whether a token usually contains semantically dependent information, i.e, encodes truthful semantic dependencies in the final layer. Specifically, our method investigates if tokens encode more semantic information from semantically related words compared to unrelated words in the sequence. We find that this holds for most tokens.
RoBERTa (encoder only)
91.06
TiayRoBERIa (eacoder only)
91.29
96.40
ALBERT (encuder only)
97.01
97.74
DistiBERT (cacoder only)
95.11
96.06
DeBERTa (encoder only)
99.62
09.74
MobileBERT (cucoder only)
96/49
99.37
Minil M (encoder only)
88.00
93.42
GPT:2 (decoder-only, auto regressis))))
75.15
100.00
1Ma (decoder-only, auto-regressive)
95.50
100.00
we calculate the percentage P that the i-th token's perturba- tion in input layer primarily affects its corresponding output token of in a transformer-basse model fe on M tested token
cases follows:
To assess whether token effectively encodes truthful se- mantic dependency. we first randomly select a word w.. We then identify group Goo containing the indices of semanti- cally dependent tokens by leveraging semantic dependency parsing tools SpaCy (Honnibal et al., 2020), which parse the words in the sentence that are semantically dependent with w?, including both head and children in parsing tree and the word itself. SpaCy leverages pre-trained neural network model to predict syntactic relationships between words, offering more comprehensive annotations than hu- man labeling. Next, we estimate a semantically dependent token group G2p by changing x! and obtain the indices of top Kopp tokens most sensitive to the change of z!. Finally, we evaluate the alignment between Gep and E2P-
P(fo)
1 M
Σ
1 (t=arg max Auf u²?
For the second validation, we calculate the percentage of cases when each input token z! affect each output token 2/. i.c., Aef kp > O, for all i.j NV
Experiments We validate model's self-information reten- tion and sequence-level semantic aggregation using vari- ous sentences from six datasets, including gsm8k (Cobbe et al., 2021), Yelp (Zhang et al., 2015), GLUE (Wang et al., 2019), CNN//ailyMail (Hermann et al., 2015), OpenOrca (Lian et al., 2023) and WikiText (Merity et al., 2016). For each model, over 100,000 token cases were evaluated for each datasets (each token perturbation is treated as one case, 600,000 cases in total). Our analysis involves 10 var- ious Transformer- et al., 2018). RoBERTa (Liu, 2019), ALBERT (Lan, 2019), DistilBERT(Sanh, 2019), DeBERTa (He et al., 2020), Mo- bileBERT (Sun al., 2020), MiniLM (Wang et al., 2020), GPT (Radford et al., 2019), and LLaMA (Touvron et al., 2023). Noted that we compute changes for nearly all to- kens (over 95%) in each sequence, excluding special tokens such as [CLS] and [SEP], which ensures a comprehensive assessment of the semantic dependency across the input.
Semantically Dependent Token Groups A group G2 contains the indices of tokens semantically dependent on z.. To identify a semantically dependent token group Gug- we leverage existing semantic dependency parsing methods to obtain the semantically dependent word group Ww: of the word w!, then convert it into a token group I. Intu- itively, dependency parsing analyzes the grammatical struc- ture of a sentence, establishing relationships between "head" words and the words that modify them. For example, in the sentence "The quick brown fox jumps over the lazy dog.". the word "fox" is semantically related to word "quick", "brown" and "jumps" based on their grammatical dependen- cies. Once W is identified, each word w, in Ww! is converted into its corresponding token indices, and wi is converted into 2P. forming Gog-
models, including BERT (Devlin
The results in Table summarize two key metrics: the first column represents the percentage of tokens that pri- marily retain their original semantic information, while the second column indicates the percentage of input tokens that propagate semantic information to other tokens. Compared to BERT and ILaMA, there is a part of tokens that do not preliminarily retain their original information in GPT. From this experiment, we can observe that most tokens primarily retain their original semantic information, even as they pass through the transformer layers. Additionally, we verify that
Results
Estimated Semantically Dependent Token Group by Leveraging Token Perturbation To estimate the seman- tically dependent token group G2e for each token zQ. we
'In our experiments, we do not consider the case when w. converted to subword tokens.
5

---

A Lens into Interpretable Transformer Mistakes via Semantic Dependency
Table 2: Alignment scores that indicate how well individual tokens encode truthful semantic dependencies (%).
otherwise, the resulting content may be random or confus- ing. Although Section 4 showed that transformer models can encode truthful semantic dependencies in the final layer, they still produce incorrect outputs in certain contexts. This suggests that correctly encoding most semantic dependen- cies is insufficient to prevent mistakes. We hypothesize that such mistakes arise from the model's tendency to encode false semantic dependencies in tokens through transformer layers. Intuitively, in the final layer, token representations are transformed via a linear prediction layer to produce out- put logits. However, the limited discriminative power of this linear layer makes it susceptible to errors when tokens encode false semantic dependencies from other unrelated or misleading tokens, ultimately leading to incorrect pre- dictions. To test our hypothesis, we conduct an empirical analysis using the question answering (QA) task, which is particularly suitable for evaluating the influence of token- level semantic dependencies because QA inherently requires the model to understand and associate tokens in a question with those in the context.
Model
Average Alignment Score (%)
BERT
87.86
RoBERTa
87.71
TonRRBERRTa
82.44
ALBERT
88.77
DistiBERT
88.88
DeBERTa
87.17
MobilcBERT
85.80
MiniLM
84.62
GPT-2
93.41
LLaMA3
92.47
measure semantic dependency score A 124 by Eq. (5) for each token 2/ in the final layer L.. Then we rank it and select the largest Koop indices within the sequence into a set denoted as Czg-
C. - -i |.j E indices of max K.p (A2z 20-5 j- = L.......... N)). (7)
Calculating Alignment Score To assess the alignment between our estimated semantically dependent token group T. and the semantically related token group Gag. we com- pute the alignment score S, to measure the overlap between Gog and Gzg:
In this section, we firstly identify that model mistakes in QA tasks stem from incorrect semantic dependencies en- coded in question tokens. Specifically, our method exam- ines whether the semantic dependency strength between wrong answer tokens and question tokens exceeds that between correct answer tokens and question tokens. Fur- thermore, to localize model parameters that encode semantic dependency, we propose a method to pinpoint group of attention head parameters responsible for token-level se- mantic dependency. We demonstrate that directly pruning the parameters to correct these mistakes is challenging, as the same parameters may encode both correct and incorrect semantic dependencies.
Gey G
Sap
(8)
Kiop
where Gec G represents the number of overlapping tokens between G. and Gzp. A high alignment score means the tokens influenced by the perturbation of z tend to be the ones that are semantically related to it, indicating models' ability to encode truthful semantic dependency.
score
5.1. Evaluation of Correct & False Semantic Dependencies in QA Task
We conducted this experiment on 10 transformer models. We first construct a specialized word dependency dataset using SpaCy. This dataset includes sentences from the GLUE dataset, where each word (as one case) in the sentence is annotated with its semantically dependent word groups as standard dependency data. For each model, we evaluated over 10,000 cases, where each case corresponds to perturbing a single token and computing the alignment score. The average alignment scores across cases are presented in Table 2. The overall high alignment scores indicate these models can generally encode truthful semantic dependencies in final-layer tokens.
Experiments and Results
To test our hypothesis that model errors often result from falsely aggregated independent semantic information within tokens, we analyze QA pairs where the language model outputs either the correct answer extracted from the context or an incorrect one. We then compare the semantic depen- dencies between tokens in incorrect answers and question tokens against those in correct answers within a question- answering (QA) task.
Consider the QA example illustrated in Figure 4, where the context provides the correct answer "national anthem" and an misleading phrase "sign language." If the BERT model incorrectly outputs "sign language" instead of "na- tional anthem", this presents opportunity to examine the underlying semantic dependencies between context tokens and question tokens that led to the mistake.
5. When the Model Makes Mistakes, It Falsely Aggregates Semantically Independent Information within a Token
Models rely on correctly encoding semantic dependencies to generate coherent and contextually appropriate outputs;
Formally, let Q {q.)) No represent the set of tokens in
6

---

A Lens into Interpretable Transformer Mistakes via Semantic Dependency
Question:
What
did
Mary
translate
Deperdency Hestre
athem
languange
semantic dependency score Agia from context token to question token
4 2.13
Gas-eann Head
national
1,000
A 1.38 Hee
1,00
Context:
0206
The
famous
singer
performed
the
national
anthem
A4t81900 Legar
6000
correct answer
correct semantic dependency: "unthem" to "Mary"
wrong scmantic dependency: "siga" to "TT"
while
Mary
provided
sign
language
translation
wrong model answer
Contribution heatmap of attention head group in BERT for correct and wrong semantic dependency
Semantic dependency in question-answer instance when model output wrong
Figure 4: A question-answer instance for false semantically dependent information within tokens.
the question, Acorect = (a.), Ne; represent the correct an- swer tokens in the context, and Awrong {aî /==1 represent the incorrect answer tokens in the context. For each answer token& a, we measure its semantic dependency on each ques- tion token qj Q by computing a semantic dependency score affla I by Eq. (5). This score quantifies the degree to which answer token a, influences the question token qj in the final layer L of the model. Next, we determine the max- imum semantic dependency score for each answer token by selecting the highest Aafla? across all question tokens
Table 3: Two-by-two possibility table for model answer correctness and semantic dependency correctness.
Nw
Correct Dependency
Incorrect Dependency
Answer Correctly
P'
1 ('))))
Answer Incorrectly
1 P(fe)
Pifo)
For each QA instance, we first determine whether the model fails to output the correct answer by evaluating the F1 score between the model's predicted answer and the ground truth answer. We consider a prediction to be incorrect if the F1 score is below 0.6. Consequently, we collect these incor- rect answer cases (where F1 < 0.6) for further analysis to examine the presence of false dependencies. This selection criterion ensures that we focus on substantial mistakes rather than minor discrepancies, thereby providing robust basis for evaluating semantic dependency misalignments.
No A l max j=1 An.
For both correct and incorrect answers, compute the highest dependency scores across all answer tokens:
Nc max A &
A Aunogl
Nw max A ag.
Accmect12
In these selected failed QA cases, we compare the semantic dependencies between question tokens and correct/incorrect answer tokens. For each case, calculate whether the max- imum semantic dependency score of incorrect answer tokens Auungii exceeds that of correct answer tokens A'A A'Aums o' This comparison allows to assess whether the model's mistakes are associated with false semantic dependencies from incorrect context tokens influencing question tokens. For successful QA cases, we calculate the percentage P'(fo) when the maximum semantic dependency score of correct answer tokens A' A'A exceeds that of incorrect answer tokens, A'A where the incorrect tokens are randomly sampled T times from the remaining context tokens (ex- cluding the correct answer tokens). Finally, we summarize P(fe) and P' (fo) for all models.
To evaluate whether the maximum dependency score for incorrect answers exceeds that of correct answers when model makes mistakes, we calculate the percentage that A o is greater than Aormi1o given the question and answer pairs where the model makes mistakes. Specifically,
P(fo) = H
1
Σ
1
(10)
Q
i=1
where H represents the total number of failed QA instances.
Experiments We apply our evaluation method to the Stan- ford Question Answering Dataset (SQuAD) 1.1 (Rajpurkar al., 2016), which comprises context paragraphs extracted from Wikipedia articles, along with manually crafted ques- tions and their corresponding correct answers. Each QA instance in the dataset provides context from which the correct answer is continuous span of text, which means the answer exists verbatim in the context. Our analysis in- volves processing over 100,000 QA validation cases across 10 Transformer-basee models.
For each model, we provide a two-by-two possi- bility table (shown in Table 3) for model answer correctness and semantic dependency correctness. P(fe) stands for the percentage when the model answers incorrectly and seman- dependency is incorrectly encoded. P'(fo) stands for
Results

---

A Lens into Interpretable Transformer Mistakes via Semantic Dependency
Table 4: Summarized percentages for two-by-two possibility table and Fl scores of all models. Results based on GPT evaluations are provided in the Appendix A.3.
BERT
RoBERTa
tiny iinyRoBERTa
ALBERT
DistiBBRRT
DeBERTa
MobileBERT
MiniLM
GPT-2
ILaMA3
P(fo)
79.07
69.20
77.94
71.86
81.80
75.32
66.61
T7.36
48.04
64.56
1- P(fo)
20.93
30.80
22.06
28.1
18.20
24.68
33.39
22.44
51.90
35.44
P'(fa)
93.26
82.32
83,33
87.05
96.48
89.25
75.24
91.97
81.25
70.56
PP)'())
6.74
17.68
16.67
12.95
3.52
10.75
24.76
8.03
18.75
29.44
Average F1 Score (%)
92.93
84.86
82.83
80.56
85.71
91.69
81.19
85.34
0.78
35.81
the percentage when the model answers correctly and the semantic dependency is correctly encoded.
encode semantic dependencies. Our motivation stems from two key considerations: 1). If correct and incorrect de- pendencies are controlled by different parameters, we can directly modify the ones responsible for incorrect dependen- cies (e.g. pruning methods) to improve model performance. 2). Conversely, if they share the same parameters, we need other strategies to address the challenge of removing false dependencies without affecting the correct ones.
The final results for all QA cases and average F1 score are summarized in Table 4, which generally shows significant proportion of model mistake cases across various models can be attributed to falsely encoded semantic dependencies. For instance, in BERT's case, the high percentage P(fs) implies that when the model selects an incorrect answer, it is more likely due to the erroneous answer tokens causing stronger semantic influence on the question tokens than the correct answer tokens. Conversely, the overall high (S() suggests when the model correctly encodes the semantic dependency in the final-layer token, it usually provides the correct answer. These findings highlight the importance of semantic dependency encoded in the final-layer token for model predictions.
As discussed above, the attention mechanism plays key role in propagating semantic information between tokens, ultimately enabling the final layer token to encode various semantic dependencies. Building on these foundations, we focus primarily on analyzing the contribution of attention head parameters in this paper. Specifically, we propose a method identify attention heads primarily responsible for encoding specific token dependencies.
The variation in Percentage across different models high- lights inherent differences in how each architecture manages semantic dependencies and mitigates the impact of mislead- ing information. For models like DistilBERT and BERT, the higher P(fe) suggests that their architecture may be more susceptible to false dependencies when mistakes occur. For models like RoBERTa and MoBileBERT, the Lower P(fs) means false dependency accounts for a small proportion in failed QA instances, which means there may be some other factors that lead to wrong outputs. Unlike the BERT series, GPT-2 and LLaMA show much lower P(Jo) and F1 scores. This discrepancy can be attributed to their generative nature where they tend to produce new words synonyms rather than reproducing original ground-truth answers. Their logic of answer generation differs from that of the BERT-based QA models, potentially introducing inconsistencies in how semantic dependencies are encoded and impacting semantic dependency evaluation.
Inspired by previous study (Gandelsman et al.. 2024), the contribution of 1-th MHA on j-th token can be broken down into tokens and heads.
H
MHA/
Lh Fi
Lhh 2
of WoR
w. 1-1
1=1
(1))
Specifically, for any token dependency, i..., token depen- dency from i-th token to j-th token, including correct or wrong token dependency in QA tasks mentioned above, we replace the i-th the token with K randomly sampled tokens. Then we measure each head's contribution to semantic de- pendency by calculating average change A I//la between original head contribution and perturbed head contributions:
A 2/14?
1 K
Σ
(.(K))
x
(.h(org)
(12)
5.2. Localize Parameters of Attention Head Group Responsible for Semantic Dependency
As shown in Figure 4(b), we test the semantic dependency contribution score A la? of each attention head in BERT for both wrong semantic dependency between "sign" and "?" and correct semantic dependency between "anthem" to "Mary" in corresponding QA instance in Figure 4(a). The heatmap reveals the group of attention heads (highlighted in brighter colors) that mutually contribute to a semantic dependency in this specific context.
As shown above, mistakes in QA tasks are closely tied to token-level semantic dependency. To better understand the network's role in model mistakes and evaluate whether false semantic dependencies can be adjusted to improve model performance, we focus on localizing the parameters that
8

---

A Lens into Interpretable Transformer Mistakes via Semantic Dependency
Experiments To evaluate how parameters in attention heads contribute to semantic dependencies across various cases, we conducted experiments on different fine-tuned models commonly used in QA tasks with high accuracy (F1 > 0.8). Specifically, we analyzed the frequency of each top 5% contributing attention head for correct and false semantic dependency across all failed QA cases (shown in Figure 5). Due to space constraints, we present the re- sults from the four perforning models and include additional experiments in Appendix A.4.
to encoding correct semantic dependencies and indiscrimi- nate removal of attention heads may inadvertently disrupt essential task-specific dependencies. Our finding highlights critical technical challenge: how to disentangle and opti- mize attention mechanisms to suppress false dependencies while preserving correct ones. Future work may require more targeted re-weighting or regularization strategies to achieve this balance.
6. Discussion and Future Work
Our current method has certain limitations, which we be- lieve present valuable opportunities for future work. Firstly, our analysis relies on perturbation-base approaches to as- sess token dependencies, which require that answer tokens appear in the context. This limits its applicability to scenar- ios where the model generates answers not directly found in the input. We aim to expand our ability to effectively analyze dependencies in such cases.
Nor holiialdd hrquincy
0.6
0.5 Sagaae
Ttard
6.2
02
12
14
10
0,1
0,7
Ro
homalloo Fracuiicy
03.00 Aennalie Freciuncs
Additionally, perturbation involves removing existing infor- mation and introducing new information, which can cause variability in output tokens. For instance, replacing a token with a semantically similar yet different token may lead to significant variation depending tion. We address this by randomly sampling new tokens to ensure diversity and reduce bias, though some variability remains. Future work will focus on refining this calibration.
Heard
0. 0:
14
10
15
20
0.0
0.0
the model's interpreta-
(a) BERT
(b) DistilBERT
Dorma Soad Fricuaani
Nermal Dormaliooe Freelincy
Had
Dhard
Our analysis primarily focuses on the semantic dependen- cies between final-layer tokens and first-layer tokens. This design choice is motivated by our goal of understanding errors in the model's output, where the final-layer token representations are expected to have the most direct influ- ence, as supported by prior studies. While our current study centers on the final layer, our method is general and can be applied to intermediate layers as well. We will explore token dependencies across different layers in future work.
10
0
08
10
20
10
0000
Lager
Layar
Marmaloud moqiaroo
Rormalled Sedmalledduuuuuuee
100
22
10
10
Layer
Layor
7. Conclusion
(c) DeBERTa
(d) MiniLM
In this paper, we delved into the internal mechanisms of transformer models to explore how semantic dependencies are encoded in tokens, which can contribute the mis- takes produced by language models. Extensive experiments reveal that: 1) most tokens primarily retain their original semantic information across layers. 2) models can encode truthful semantic dependencies in final-layer tokens. and 3) model mistakes often stem from tokens encoding incorrect dependencies. However, shared attention head parameters help encode both correct and false dependencies, indicat- ing the challenge of removing incorrect dependencies. We believe these insights can offer valuable implications for future transformer model design.
Figure 5: Frequency of each top 5% contributing attention head for correct (green) and false (red) semantic dependency across all failed QA cases.
Results Figure 5 reveals shared group of top attention heads responsible for both correct and false semantic depen- dency in these models. This suggests that model mistakes in QA tasks stem from the parameters of these specific attention heads. However, it also suggests false seman- tic dependencies cannot be reduced by directly disabling them (e.g., via head pruning approaches (Voita et al., 2019; Michel et al., 2019)) because these heads also contribute

---

A Lens into Interpretable Transformer Mistakes via Semantic Dependency
Impact Statement
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. CoRR, abs/21 10.110..111 14168, 2021.
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: pre-training of deep bidirectional transformers for lan- guage understanding. CoRR, abs/1810.04805, 2018.
Acknowledgements
TL is partially supported by the following Australian Re- search Council projects: F1220100318, DP220102121, LP220100527, LP220200949, and IC190100031. BH was supported by RGC Young Collaborative Research Grant No. C2005-24Y and NSFC General Program No. 62376235.
Dozat, T. and Manning, C. D. Simpler but more ac- curate semantic dependency parsing. arXiv preprint arXiv: 11X7,00007. 2018.
Eberle, O., Büttner, J., Kräutli, F., Müller, K.-R., Valleriani, M., and Montavon, G. Building and interpreting deep similarity models. IEEE Transactions on Puttern Analysis and Machine Intelligence, 44(3):1149-116 2020.
References
Abnar, S. and Zuidema, W. Quantifying attention flow in transformers. arXiv preprint 2005:2005,000228 2020.
Elhoushi, M., Shrivastava, A., Liskovich, D., Hosmer, B., Wasti, B., Lai, L., Mahmoud, A., Acun, B., Agarwal, S., Roman. A.. et al. Layer skip: Enabling early exit inference and self-speculative decoding. arXiv preprint arXiv:2404.1677 16710, 2024.
Ali, A., Schnake, T., Eberle, O., Montavon, G., Müller, K.-R., and Wolf, L. Xai for transformers: Better explana- tions through conservative propagation. In International conference on machine learning, pp. 435-451. PMLR, 2022.
Feng. J. and Steinhardt, J. How do language models bind entities in context? arXiv preprint 1rXiv:2310.17191 2023.
Amini, A., Pimentel, T., Meister, C., and Cotterell, R. Natu- ralistic causal probing for morpho-syntax. Transactions of the Association for Computational Linguistics, 11:384- 403, 2023.
Finlayson, M., Mueller, A., Gehrmann, S., Shieber, S., Linzen, T., and Belinkov, Y. Causal analysis of syn- tactic agreement mechanisms in neural language models. arXiv preprint arXiv:2 06087, 2021.
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C., al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.0000 2022.
Gandelsman, Y., Efros, A., and Steinhardt, Interpret- ing clip's s image representation via text-based decomposi- tion, 2024. httLtttss///axtt org/ ab3/1 05916.
Björkelund, A., Bohnet, B., Hafdell, L., and Nugues, A perffrreffem syntactic and semantic dependency parser. In Coling 2010: Demonstrations, pp. 33-36, 2010.
Ganguli, D., Lovitt, L.., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.0788 2022.
Brown, T. B. Language models are few-shot learners. arXiv preprint arXiv:2005.14411 2020.
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert- Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633-2650, 2021.
Gehman, S., Gururangan, S., Sap, M., Choi, Y., and Smith, N. A. Realtoxicityproppss Evaluating neural toxic de- generation in language models. In EMNLP (Findings), volume EMNLP 2020 of Findings of ACL, pp. 3356-3369, Association for Computational Linguistics, 2020.
Chakraborty, A., Alam, M., Dey, V., Chattopadhyay, A., and Mukhopadhyay. D. Adversarial attacks and defences: A survey. CoRR, 0bs/1810000069, 2018.
Geva, M., Bastings, J., Filippova, K., and Globerson, A. Dis- secting recall of factual associations in auto-regressive language models. arXiv preprint arXiv:2304. 147677 2023.
Chen, H., Zhang, M., Li, J., Zhang, M., Øvrelid, L., Hajič, J., and Fei, H. Semantic role labeling: A systematical survey. arXiv preprint arxiv:2502 08660, 2025.
He, L., Lee, K., Lewis, M., and Zettlemoyer, L. Deep se- mantic role labeling: What works and what's next. In
10

---

A Lens into Interpretable Transformer Mistakes via Semantic Dependency
Proceedings of the 55th Annual Meeting of the Associ- ation for Computatioonal Linguistics (Volume I: Long Papers), pp. 473-483, 2017.
Lian, W., Goodson, B., Pentland, E., Cook, A., Vong, C., and "Teknium". Openorea: An open dataset of gpt augmented flan reasoning traces. https: //ttps::
/ /huggingface..ecmc///////////ace 2023.
He, P., Liu, X., Gao, J., and Chen, W. Deberta: Decoding- enhanced bert with disentangled attention. arXiv preprint arXiv:2006.0366 03.544 2020.
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. Holistic evaluation of language models, arXiv preprint arXiv:221.099 10, 2022.
Hermann, K. M., Kociský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., and Blunsom, P. Teaching machines to read and comprehend. In NIPS, pp. 1693- 1701, 2015.
Liao, K., Zhang, Y., Ren, X., Su, Q., Sun, X., and He, A global past-future early exit method for accelerating inference of pre- trained language models. In Proceedings of the 2021 conference of the north american chupter of the association for computational linguistics: Human language technologies, pp. 2013-2023, 2021.
Hewitt, J. and Manning, C. D. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume I (Long and Short Papers), pp. 1229-138,, 2019.
Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv: 100.00990079 2021.
Honnibal, M., Montani, I., Landeghem, S. V., and Boyd, A. spaCy: Industriall-strength Natural Language Processing in Python. attps://spat
io, 2020. Version 3.0.
Liu, Y. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 364, 2019.
Janizek, D., Sturmfels, P., and Lee, S.-I. Explaining explanations: Axiomatic feature interactions for deep networks. Journal of Machine Learning Research, 22 1104):1-54, 2021.
Mel'čuk, I. Language: Dependency. Elsevier, 2001.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models, 2016.
Ji, Z., Lee, N., Frieske, R., Yu, T. Su, D., Xu, Y., Ishii, E., Bang. Y. J., Madotto, A., and Fung. P. Survey of halluci- nation in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
Michel, P., Levy, O., and Neubig, G. Are sixteen heads really better than one? Advances in neural information processing systems, 32, 2019.
Navigli, R., Conia, S., and Ross, B. Biases in large language models: origins, inventory, and discussion. ACM Journal of Data and Information Quality, 15(2):1-21, 20233
Jia, R. and Liang, P. Adversarial examples for evalu- ating reading comprehension systems. arXiv preprinl arXiv:170.007328, 2017.
OpenAL. GPT-4 technical report. CoRR, abs/2303.08774 2023.
Kang, D., Li, X., Stoica, I., Guestrin, C., Zaharia, M., and Hashimoto, T. Exploiting programmatic behavior of lmms: Dual-use through standard security attacks. In 2024 IEEE Security and Privacy Workshops (SPW), pp- 143. IEEE, 2024.
Ousidhoum, N., Zhao, X., Fang, T., Song, Y., and Yeung, D.-Y. Probing toxic content in large pre-trained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4262-4274, 2021.
Khashabi, D., Khot, T., Sabharwal, A., and Roth, D. Ques- tion answering as global reasoning over semantic abstrac- tions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Pimentel, T. Valvoda, J., Maudslay, R. H., Zmigrod, R., Williams, A., and Cotterell, R. Information- theoretic probing for linguistic structure. arXiv preprint arXiv:2004.0300 2020.
Lan, Z. Albert: A lite bert for self-supervised learning of lan- guage representations. arXiv preprint arXiv:1909.1 1194, 2019.
Li, X., Roth, D., and Small, K. The role of semantic infor- mation learning question classifiers. In Proceedings of the International Joint Conference on Natural Language Processing, 2004.
Qi, P.. Zhang, Y., Zhang, Y., Bolton, J., and Manning. C. D. Stanza: A Python natural language process- ing toolkit for many human languages. In Proceed- ings of the 58th Annual Meeting of the Association

---

A Lens into Interpretable Transformer Mistakes via Semantic Dependency
for Computational Linguisties: Linguattio System Demonstrations, 2020. https:/::/ stwanorrd...oomo. wwd./pub// qi202Ostanza.por
Taori, R. and Hashimoto, T. Data feedback loops: Model-
driven amplification of dataset biases. In International Conference on Machine Learning, pp. 33883-33920. PMLR, 2023.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1())9, 2019.
Tenney, I. Bert rediscovers the classical nlp pipeline. arXiv preprint
2019.
Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. SQuAD: 100,000+ questions for machine comprehension of text. In Su, J., Duh, K., and Carreras, X. (eds.), Proceed- ings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/w1/D16-- 1264. URL https: // //aclanthology wtgg//6-1//0
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models. CoRR, abs/2302.139711
Vasileiou, A. and Eberle, O. Explaining text similarity in transformer models. arXiv preprint urXiv:2405. 066006 2024.
Rogers, A., Kovaleva, O., and Rumshisky, A. A primer in bertology: What we know about how bert works. Trans- actions of the Association for Computational Linguistics, 8:842-866. 866, 2021.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In NIPS, pp. 6008, 2017.
Sanh, V. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint 1rXiv:1910.01108, 2019.
Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., and Shieber, S. Investigating gender bias in language models using causal mediation analysis. Ad- vances in neural information processing systems, 33: 12388-12400 2020.
Scheurer, J., Balesni, M., and Hobbhahn, M. Large lan- guage models can strategically deceive their users when put under pressure. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024.
Voita, E., Talbot, D., Moiseev, F., Sennrich, R., and Titov, I. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv: /905.09418. 2019.
Schnake, T., Eberle, O., Lederer, J., Nakajima, S., Schütt, K. T., Müller, K.-R., and Montavon, G. Higher-order explanations of graph neural networks via relevant walks. IEEE transactions on pattern analysis and machine intel- ligence, 44(11):5581-756, 2021.
Wang. A., Singh, A., Michael, J.. Hill, F., Levy, O., and Bowman, S. R. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR. OpenReview.net, 2019.
Schuster, T., Fisch, A., Gupta, J.. Dehghani, M., Bahri, D., Tran, V., Tay, Y., and Metzler, D. Confident adaptive language modeling. Advances in Neural Information Processing Systems, 35: 15:17456-17472. 2022.
Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., and Zhou, M. Minilm: Deep self-attention distillation for task agnostti compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33: 5776-5788 2020.
Shen, D. and Lapata, M. Using semantic roles to improve question answering. In Proceedings of the 2007 joint conference empirical methods in natural language processing and computational natural language learning (EMNLP-CONLL), pp. 12-21, 2007.
Wattenberg, M. and Viégas, F. B. Relational composition in neural networks: A survey and call to action. arXiv preprint arXiv:2407. /4662, 2024.
Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E. H., Schärli, N., and Zhou, D. Large language models can be easily distracted by irrelevant context. In Inter- national Conference on Machine Learning, pp. 31210- 31227. PMLR, 2023.
Wei, A., Haghtalab, N., and Steinhardt, J. Jailbroken: How does Ilm safety training fail? Advances Neural Infor- mation Processing Systems, 36, 2024.
Weidinger, I.., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S,, Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359, 2021.
Sun, Z., Yu, H., Song, X., Liu, R., Yang, Y., and Zhou, D. Mobilebert: a compact task agnostic bert for resource- limited devices. arXiv preprint 200:20004.02984 2020.
12

---

A Lens into Interpretable Transformer Mistakes via Semantic Dependency
Wu, W. Wang, Y., Xiao, G., Peng, H., and Fu, Y. Re- trieval head mechanistically explains long-context factu- ality, arXiv preprint 155744 2024.
Wu, Z.,, Chen, Y., Kao, B., and Liu, Q. Perturbed masking: Parameter-free probing for analyzing and interpreting bert. arXiv preprint arXiv:2004. 14786, 2020.
Zhang, W. E... Sheng. Q. Z., Alhazmi, A., and Li, C. Adver- sarial attacks on deep-learning models in natural language processing: A survey, ACM Transactions on Intelligent Systems and Technology (TIST), 11(3):1-41, 2020.
Zhang, X., Zhao, J., and LeCun, Y. Character-level convolu- tional networks for text classification. Advances in neural information processing systems, 28, 2015.
13

---

A Lens into Interpretable Transformer Mistakes via Semantic Dependency
A. Appendix
A.1. Detailed Related Works
Semantic Dependency Parsing and Semantic Role Labeling. Semantic dependency parsing (SDP) (Björkelund et al., 2010; Dozat & Manning, 2018) aims to identify semantic relationships between words in a sentence, by constructing a directed graph, where nodes represent words and edges capture their semantic dependencies. Closely related to SDP, semantic role labeling (SRL) (He al., 2017; Chen et al.. 2025) focuses on identifying the predicate-argumeet structure of a sentence by assigning roles to words or phrases based on their semantic relationship a verb or predicate. Notably, studies (Li et al., 2004; Shen & Lapata, 2007; Khashabi et al., 2018) have shown that incorporating semantic role information enhances question-answering systems. However, despite their advancements, the internal mechanisms by which transformer models encode, propagate, and utilize semantic dependencies remain largely opaque. Our work bridges this gap by exploring how internal mechanisms contribute to semantic dependency encoding and how these insights can be leveraged to address mistakes.
Semantic Information Flow in Transformer. Existing work (Liao et al., 2021: Schuster et al., 2022; Elhoushi et al., 2024) have studied model activation stability in later layers of transformer models. Specifically, additional layers may contribute minimally to the refinement of token representations, which enables techniques like early exit to accelerate inference. However, whether the token in the last layer mostly contains its original semantic information in the input layer has not been studied. Previous study (Geva et al., 2023) analyzes how factual associations recalled in auto-regressive language models, highlighting the roles of MLP sublayers in enriching subject representations and attention heads in extracting attributes. Our study addresses a gap by studying how semantic dependencies are encoded in tokens and influence QA tasks in both non-auto-regressive (BERT series) and auto-regressive models (GPT, LLaMA).
Interpretable Model Mistake Based on Attention Heads. Existing works have studied specific roles of attention heads to explain model mistakes. Study (Wu et al., 2024) identifies specific attention heads, termed retrieval heads, which are critical for retrieving factual information from long contexts. The absence or malfunctioning of these retrieval heads may lead to model mistakes. Another study (Gandelsman et al., 2024) shows some attention heads CLIP have property-specific roles (e.g., location or shape), which are important for model performance. Our work offers another perspective by interpreting model mistakes via token-level semantic dependency encoding, which provides insights into understanding and correcting model mistakes under specific question -atseeringn cases. Additionally, our finding in mutual attention heads responsible for key dependencies in QA tasks also shows the importance of adjusting parameters in attention heads without tempering the correct semantic dependency encoding.
Probing Study for Linguistic Properties in Transformer. Probing methods (Rogers et al., 2021) are widely used to analyze the internal representations of pre-trained language models to determine whether specific linguistic properties are encoded. A previous study demonstrated that BERT encodes syntactic tree structures in its vector space, allowing a probing classifier to reconstruct syntactic distances between words using linear transformations (Hewitt & Manning, 2019). Additionally, the study revealed that BERT encodes high-level linguistic features like entity types, semantic roles, and relations through probing tasks (Tenney, 2019). Moreover, existing research utilized information-theone probing methods to quantify the mutual information between model representations and linguistic properties, reducing over-interpretation risks (Pimentel et al., 2020).
Token ablation is widely used, parameter-free free probing technique. For example, researchers have studied the influence of syntactic subtree structures on masked language model (ML.) predictions through ablations (Wu et al., 2020). Others analyze syntactic agreement in language models through causal interventions, identifying key neurons and attention heads (Finlayson et al., 2021). Gender bias has been investigated using causal mediation analysis (Vig et al., 2020). Naturally occurring perturbations, which refer to sentences differing in specific morpho-syntactio features, have been used to probe causal relationships (Amini et al., 2023).
These works primarily investigate how models encode syntactic and high-level semantic features, such as entity relations or syntactic structures, In contrast, our study focuses specifically on token-level semantic dependency encoding, analyzing fine-grained interactions between individual tokens rather than task-specific feature aggregation or high-level semantic encoding. Moreover, we introduce an evaluation framework to measure semantic dependency strength between two tokens without relying on prior knowledge. Our approach also identifies false semantic dependencies that arise when the model
14

---

A Lens into Interpretable Transformer Mistakes via Semantic Dependency
produces incorrect answers. Unlike static syntactic or semantic structures, our framework captures the dynamic and context-sensitive semantic dependencies, which can vary irregularly across diverse scenarios.
Feature Attribution and Binding Study. Feature attribution methods primarily aim to assess the importance of individual tokens or features to the model's output. For example, prior work on attention flow (Abnar & Zuidema, 2020) quantifies token importance through accumulated attention matrices. Gradient-base techniques like Conservative Propagation (Ali et al., 2022) is used to assess token attribution. Our study shifts focus from token importance to the semantic dependencies encoded in token representations and how these affect model behavior.
Existing semantic dependency methods based on feature/token interactions (Eberle al., 2020; Janizek et al., 2021; Schnake et al., 2021) mainly focus on studying the contribution of combinations of features or tokens to model predictions. Meanwhile, feature binding methods (Feng& & Steinhardt, 2023; Vasileiou & Eberle, 2024; Wattenberg & Viégas, 2024) often do not test whether the model's most confident output reflects encoded semantic dependencies; rather, many assume this relationship holds and study downstream properties. In contrast, our method is designed to explicitly test the assumption by evaluating whether there a dependence between the model's s output and the semantic dependency encoded in the final-layer token.
A.2. Extra Finding: The Semantic Dependency Encoded in a Token Is Influenced by Both Irrelevant Context Changes and Order of Contexts
In this section, we study whether the rank of semantic dependency strength encoded in a token changes when adding irrelevant context or simply changing the order of the context sequence. We also found some interesting phenomena after experiments: 1). Semantically related tokens remain relatively stable when altering irrelevant context or the order of the context. 2). Left context change usually causes greater influence than right context change.
Robustness studies have demonstrated that the inclusion of irrelevant context (Shi et al., 2023) or adversarial sentences (Jia & Liang. 2017) in prompts can lead to a significant decline in model accuracy. They usually work by analyzing model performance on various types of adversarial examples and attribute the decline to broader issues, such as the model's tendency to rely on surface-level features like word overlap and positional cues. To further explore the underlying reason for such performance decline from a token-level perspective, we test whether altering the irrelevant context or rearranging the order of independent sentences affects the rank of semantic dependency strength.
For example, we have two semantically independent token sequences "white rhinos are gray" and "apples are red' in Figure 6, where "apples are red" (highlighted with green background) serves irrelevant context to "white rhinos are gray". On the top half part of the figure, when we add the irrelevant context "apples are red", the rank of semantic dependency strength between the token "rhinos" and tokens in its sequence "white rhinos are gray." varied. On the bottom part of the figure, the same thing happens when we maintain the overall input semantic information unchanged and only change the order of the two token sequences. This demonstrates that even when two token sequences are semantically independent, irrelevant changes in context and the ordering of sequences can significantly alter how semantic information aggregated within each token.
Semantic Dependency Analysis with Irrelevant Context Change To validate whether irrelevant context influences the semantic dependencies of tokens in sequence, we selected two semantically independent sentences randomly sampled from dataset. Consider two sentences:
"The sky is blue." " vs "The apple is red. The sky is blue.", i.e., 81 vs (82, 81)
"The sky is blue." vs "The sky is blue. The apple is red.", i.e., 81 vs (81, 82)
We investigated whether the semantic dependencies within "The sky is blue." " remain unchanged when appended with "The apple red." on its left side or right side. Since both contexts are independent, with no semantic dependencies between them, the semantic dependencies within "The sky is blue." should remain unchanged regardless of their surrounding context in the input sequence.
Specifically, given two input token sequences are (=1)) = validate the semantic dependencies within ((()), We created two additional token sequences: ((Lef)) = (20)))), 2(+,))
( and x=(22)
respectively. Here, we |
15

---

A Lens into Interpretable Transformer Mistakes via Semantic Dependency
layer
layer
white
rhinos
whilte
grey
rhimos
layer
grey
rhinos
rhinos
grey
white
apples
are
are
red
layer
layer L
layer 0 apples
white
rhinos
rhinos
whitee
-.
are
are
grey
white rhinos
apples
rhinos
red
are
red
Figure Semantic information propagation is influenced by irrelevant context change and sequence order change.
and z R(ight)) (,))), 20(2))] where (Leff) is obtained by concatenating ( the left and ((Righ)) is obtained by concatenating (((=s) to the right. For token z from z((x), we obtain the corresponding estimated semantic dependency
token group G via Eq. (7). By using the same approach, estimated semantic dependency token groups Glef and G for
(left) and ((Rgght) can also be obtained. Then the Dependency Alteration Score (DAS) of G and G can be calculated
follows:
G (%)) = (CS(GTef G (00
(13)
where L.S(.) is the length of the longest common subsequence. In our case, it represents the longest sequence of tokens that appear in the same order in both contexts, despite irrelevant context or order changes. The score DAS
C1 measures
2 20)
how the semantic dependency changes when appending irrelevant context z ((2) to the left of the original sequence (+11),
Similar (G Right (2)) can be obtained, which measures the changes of semantic dependency when appending irrelevant context z ((z) to the right.
Semantic Dependency Analysis with Irrelevant Context Order Change For irrelevant context order change, observe whether the token dependency in sentence "The sky is blue." alters when inputting the sentence with irrelevant context order change, e.g., "The sky is blue. The apple is red." and input "The apple is red. The sky is blue... We simply use DAS( Geett (Gen (Right) to measure how the semantic dependency changes when appending the irrelevant context ((22) to the
2?
left and the right of the original sequence z 2(+1).
Experiments We conducted the semantic dependency analysis across over 5,000 cases examine the impact of irrelevant context added to both the left and right sides, as well as the effect of sequence order changes, in order to determine whether the semantic dependency encoding is context-dependent and order-dependent. Specifically, we measured the dependency ((e1) in the original sequence (ex)) This involves evaluating the dependency alterations changes when perturbing the token z; of its semantically dependent token groups by aligning the top 5 semantically dependent token groups (L 5) and by aligning all tokens from the original sequence z ((11) (L = N2). The average dependency alteration scores are presented in Figure?. 7.
Results Figure 7(a) and Figure 7(b) illustrate the changes in semantic dependency when irrelevant context is appended on the left or right side. It shows that the rank of semantic dependency strength of common token is significantly affected by the context, while relationships of semantically more related tokens (Top 5) remain relatively stable.
Figure 7(c) further compares the changes in dependency when the irrelevant context is added to the left versus the right side of the original sentence. The results reveal that adding context to the left side generally results in a greater alteration of
16

---

A Lens into Interpretable Transformer Mistakes via Semantic Dependency
Right
Rop-5
0:35
Tip Right
0.20
Stovie
0.15
0.10
0.05
Models
Modils
(a) Left context change
(b) Right context change
Right Context
0.4
Right
0.35
Crder lop Order
0.30
0.3
330000
0.15
0.10
0.85
Models
Models
(c) Left & right context
(d) Order change
Figure 7: Semantic Dependency Alteration Score when irrelevant context context order changes.
semantic dependencies compared to the right side. This suggests that the order of irrelevant context can differentially impact the model's semantic dependency structures.
Figure 7(d) demonstrates the impact of altering the sequence order semantic dependencies. The results also show that irrelevant token groups are easily influenced by unrelated contexts, while semantically more dependent tokens exhibit greater resilience to such alterations.
Overall, our findings indicate that both the introduction of irrelevant context and the modification of sequence order dramatically influence semantic dependency within sentences. These results reinforce the importance of context placement and order in shaping the semantic dependency structures learned by Transformer-bass language models.
Insight for Future Model Design Our insight can further help training or finetuning a robust language model. Intuitively, semantic dependencies between tokens should remain robust regardless of changes irrelevant contexts or the order of independent sentences. A natural thought for future work could be regulating transformer models to maintain consistent semantic dependencies despite irrelevant context variations. This may involve implementing regularization techniques that enforce stable token representations regardless of irrelevant context sequence alterations.
A.3. More Experiments
Percentage of Token Primarily Retains Its Original Semantic Information across Different Datasets in Section
3. We measure the total percentage with various sentences from six datasets, including gsm8k (Cobbe al., 2021), Yelp (Zhang al., 2015), GLUE (Wang et al., 2019), CNN/DailyMail (Hermann al., 2015), OpenOrca (Lian et al., 2023) and WikiText (Merity al., 2016). For each model, over 600,000 token cases were evaluated (each token perturbation is treated as one case). We also observed that proportion of tokens in GPT propagate semantic information mostly to its next token. Thus, We also include the percentage the token propagating semantic information to both its next token and itself for GPT on the right. The detailed result is displayed in Table 5.
To further discuss this phenomenon, we believe that one key factor is the presence of residual shortcuts, which may encourage final-layer token representations retain information from the original input. For example, in a simple one-layer model with direct shortcut from input to output, the final-layer token is likely to closely mirror its corresponding input token. However, residual connections alone do not fully explain the observed effect. To support this, we include the percentages for GPT-2, GPT-2-Large, and GPT-2-XL across different datasets in Table 6. Although these models share the same residual architecture, larger models (e.g.. GPT-2-XL) exhibit significantly stronger semantic retention (similar percentages to BERT, e.g., around 98%) at the final-layer token level than smaller models (c.g.. GPT-2). suggests that semantic retention also influenced by other factors such as model size and complexity.
17

---

A Lens into Interpretable Transformer Mistakes via Semantic Dependency
Table 5: Percentage of a token primarily retains its original semantic information.
gsm8k
Yelp
GLUE
DailyMail
OpenOrea
WikiText
BERT
99.22
98.58
98.48
98.81
98.90
98.84
RoBERTa
89.54
93.46
89.98
95.02
93.34
96.99
TinyRoBERTa
92.29
95.16
94.43
95.11
94.38
94.39
ALBERT
96.84
97.36
97.67
96.67
97.65
95.85
DistilBERT
93.84
95.27
95.84
95.70
95.54
94.49
DeBERTa
99.69
99.57
99.54
99.67
99.46
99.78
MobileBERT
94.34
96.38
93.16
97.73
98.22
99.08
MiniLM
87.21
92.16
93.25
87.54
86.38
85.58
GPT-2
75.19/88.42 96.21
77.16/89.91
77.49/92.51 94.20
73.11 73.11/85.88
69.32/8 /81688
72.31 72.31/81.46
/81.46
LLaMA3
96.68
95.85
95.78
9-1.00
Table 6: Percentage of a token primarily retains its original semantic information in GPT series.
gsm8k
Yelp
GLUE
DailyMail
OpenOrea
WikiText
Avg. Percentage
GPT-2 (124M)
75.19
77.46
77.49
73.11
69.32
72.31
75.15
GPT-2-Large (774M)
98.49
98.47
98,16
98,17
98.34
98.08
98.29
GPT-2-XL (1.5B)
98.64
98.32
97.85
97.83
97.90
97.80
98.05
Percentage of Token Propagates Semantic Information to Other Tokens in Section 3. We also observe the change of the specific input word causes influence other tokens in the final layer in experiment of Section 3. The result in all cases (each token perturbation is treated as one case, over 600,000 cases are evaluated for each model) is shown in Table 7. Even if minor, in models like BERT, DeBERTa, and MoRileBERT, the change is almost 100%, which means each token receives pieces semantic information from almost every token in the input sequence. While in auto- regresegreesive models like LLaMA or GPT, the token only influences the tokens on this token's 's right side. We observe the changes of tokens on each tokens' left side is O. We can also observe the change exists in all tokens on each token's ''s right side, which suggests each token receives pieces of semantic information from almost all tokens its left side.
Why replacing a token with random tokens to explore semantic dependency For both finding 2 and finding 3, we need to examine how semantic dependency is encoded the final layer by replacing an input token zς and observing which final-layer token (e.g., at position j) changes. If 2 and 2 exhibit strong dependency, which means the semantic information of z is encoded in the final-layer representation at position j, then replacing z! with another token ž! should cause z to change substantially. This indicates a semantic dependency between the two tokens. However, if we replace zς with a synonym (e.g.. z("), the overall semantic meaning of the sentence may remain largely unchanged, and the model may treat z! and z similarly. In this case, we may observe a minimal change at z//, making difficult conclude whether z was originally dependent on zQ. even a true dependency existed. Therefore, we use random tokens to encourage semantic independence. also important to note that random token selection may introduce out-of-domain predictions. We believe measuring sensitivity or semantic relevance through gradients could provide valuable insights and exciting directions for future work.
Why Using Neural Dependency Parsing Tool in Section 4 Noted that our analysis relies on semantic dependency data derived with SpaCy, a pretrained neural network- dependency parser. SpaCy generates syntactic dependency trees using robust neural architectures trained on large annotated corpora, offering a reliable approximation of semantic dependencies. To our knowledge, no token-level semantic dependency dataset with comprehensive human annotations exists. Constructing such dataset would be prohibitively expensive and prone to omissions due to the complexity of identifying all dependent token relationships manually. Thus, we use neural dependency parsing tool to generate a specialized semantic dependency dataset for our experiment.
We additionally conducted experiments using another widely adopted dependency parser, Stanza (Stanford NLP) (Qi et al., 2020), to validate the robustness of our findings. As shown in Table 8, the results obtained with Stanza are consistent with those derived from SpaCy, further supporting the conclusion that transformer models encode truthful semantic dependencies
18

---

A Lens into Interpretable Transformer Mistakes via Semantic Dependency
Table 7: Percentage of a token propagates semantic information to other tokens.
gsm8k
Yelp
GLUE
Dail
OpenOrea
WikiTeet
BERT
99.44
99.09
99.16
99.20
99.34
99.52
RoBERTa
92.94
95.00
90.76
95.98
95.23
98.24
TinRRBERTaa
96.46
96.42
97.04
96.42
95.98
96.07
ALBERT
97.88
97.99
98.35
97.34
98.23
96.63
DistiBBERT
95.44
95.89
96.37
96.29
96.42
95.93
DeBERTa
99.86
99.63
99.66
99.77
99.68
99.82
MobileBERT
99.43
99.22
98.96
99.38
99.50
99.74
MiniLM
94.03
95.51
94.71
91.74
92.19
92.35
GPT-2
100.00
100.00
100.00
100.00
100.00
100.00
LLaMA3
100.00
100.00
100.00
100.00
100.00
100.00
Table 8: Alignment scores indicating how well tokens encode truthful semantic dependencies using Stanza and Spacy (%).
BERT
RoBERTa
tinyRoBERTa
ALBERT DistiBBERT
DeBERTa
MobileBERT
MiniLM
GPT-2
LLaMA3
SpaCy
87.86
87.71 86.9
82.44
88.77
88.88
87.17 83.69
85.8
84.62
93,41
92.47
Stanza
84.33
81.14
85.53
87.19
80.98
83.67
91.42
90.32
in their final layers.
Note that although model-estimated semantic dependencies can be easily obtained, the main challenge is that existing semantic dependency parser methods usually cannot measure dependencies at the subword level. This makes direct comparison difficult. To address this issue, we may need to manually annotate the semantic dependencies and compare them with those estimated by the models, which is costly and hard to scale.
Why Using Longest Common Subsequence in Section A.2 Consider a simple example to understand how LCS captures changes in token order: Suppose we have two sequences, A = [1, 2, 3, 4) and B = [2, 3, 4, 1]. In moving from sequence A to sequence B, the order of the tokens changes such that the token "1" moves from the beginning to the end. Here, the LCS between A and B is the subsequence [2, 4], which has a length 3. This subsequence represents the largest set of tokens that have retained their original order between the two sequences. Since the total number of tokens, N, is 4, the LCS length of 3 indicates that one token ("1") changed its position relative to the others. By calculating DAS 0.25, find that quarter of the token order has been altered due to the change in context. Thus, lower LCS value (relative to N) results in higher DAS, reflecting a more significant change in token dependency patterns. This metric effectively highlights how sensitive the token dependencies are to contextual modifications, demonstrating the dynamic nature of semantic processing in natural language systems.
Why Choosing QA as the Primary Task for Our Experiments We chose the question-answering (QA) task because it is particularly well-suited for evaluating the impact of semantic dependency mistakes at the token level. QA tasks inherently involve understanding and associating tokens a question with those in the context, making them ideal for testing the model's ability handle complex dependencies. This directly aligns with the focus of our study, which explores how false encoded semantic dependencies lead to model mistakes. Additionally, to validate our findings, it is crucial have ground truth datasets that clearly present correct and incorrect dependencies. QA tasks provide datasets like SQuAD, where the answers are explicitly tied certain context tokens. These datasets enable to systematically evaluate how dependency mistakes between question and context tokens contribute to prediction mistakes.
Why Threshold of F1 0.6 is Chosen and Additional Evaluation Using More Advanced ChatGPT Model. The threshold of F1 < 0.6 for identifying incorrect answers was determined empirically. Since our goal is to assess whether incorrect answers are associated with incorrect semantic dependencies, an F1 score below 0.6 indicates that over 40% of the tokens predicted by the model differ from those in the original answer, which strongly suggests the answer is likely incorrect.
To further strengthen this analysis, and following existing work, we conducted additional experiments using ChatGPTT-40
19

---

A Lens into Interpretable Transformer Mistakes via Semantic Dependency
Table 9: Additional experiments ChatGPT-4o model to find incorrect cases,
BERT
RoBERTa
LnnRoBBRRTa
ALBERT
DistiBBRT
DeBERTa
MohileBERT
Minil.M
GPT-2
LlaMA3
P(fo) (FI-0.6)
79.07
69.20
77.94
71.86
81.80
75.32
66.61
77.56
48.04
64.56
FI Score
92.93
84.86
82.83
80.56
85.71
91.69
81.19
85.34
0.78
35.81
P(fo) (GPT-40 select)
79.00
68.42
73.31
66,11
81.79
77.84
68.69
77.56
59.60
62,35
Accuracy
88.45
78.00
78,00
74,63
76.63
90.44
74.50
78.90
0.10
14,68
Table 10: The score of GPT and LLama models using one-shot setting and zero-shot setting.
F1 (0-shot)
Fl (1-shot)
GPT-2 (124M)
0.78
5.5
GPr----Lrge (774M)
7.3
21.09
LLaMA3-8B-insn (8B)
35.81
76.27
models to compare the model's answer with the ground truth and find incorrect cases. The results displayed in Table 9 are similar to using Fl <0.6.
Why Choose 0-shot F1 Evaluation on LLaMA and GPT Models To ensure a fair comparison, we evaluated LLaMA and GPT models using the same zero-shot (0-shot) setting as BERT. This is the reason that they present low accuracy. Table 10 shows the F1 score using a one-shot setting, which aligns with official benchmark evaluations.
A.4. Localize Parameters of Attention Head Group Responsible for Semantic Dependency in Section 5.2
To better understand the network's ' role in model mistakes and evaluate whether false semantic dependencies can be adjusted to improve model performance, we focus on localizing the parameters that encode semantic dependencies. We have developed a method to identify the attention heads primarily responsible for specific token dependencies. Here, we present the intuition and detailed equations.
Intuitively, when the input token carrying specific semantic information changes, the attention heads relevant corresponding semantic information propagation will exhibit significant changes in their outputs, while the outputs of irrelevant heads will remain relatively unchanged. Therefore, by identifying heads with the highest variation in their contribution given token dependency, we can pinpoint the group of attention heads that are mutually responsible for any token dependency including wrong or correct token dependency in the QA task.
As mentioned in Eq. (2), transformer encoder or transformer decoder is residual network built from layers, each of which contains multi-head self- self-attention (MHA) followed by feed forward network (FFN) block.
In the 1-th MHA layer, the input stream 2 is processed separately by H attention heads. Specifically, the input sequence z is separately projected into Q, K, V matrix in l-th attention head of 1-th layer as follows:
Q h - whh.
KLh z Z'
v Z Z'i
(14)
Then attention weight matrix A h E R N N is calculated as follows:
QKT
A 1.h softmax
(15)
ViTe
The output of each attention head is
0.h = Alvel
(16)
For multi-head attention, the outputs of each head are concated and projected to Z E RND where Wo is the output weight matrix.
MHA Concat 4 0 ,...., 12. Wo
(17)
20

---

A Lens into Interpretable Transformer Mistakes via Semantic Dependency
The class token and the other tokens share the same computation process. Inspired by previous study (Gandelsman et al., 2024), the contribution of 1-th MHA on j-th token be broken down into tokens and heads. We can observe that given token, each context token contributes to this token by adding operation for semantic information aggregation, which generates context-related token representation.
H
MHAJ
ih
2ihh- 1; = ai w 1-1
(18)
Specifically, for any token dependency, i.e., token dependency from i-th token to j-th token, including correct or wrong token dependency in the QA task mentioned above, we replace the i-th token with K randomly sampled tokens. Then we measure each head's contribution to semantic dependency by calculating average change A 1//
between original head
contribution and perturbed head contributions as follows:
A 1.b 3
1 K
I.M())
3/
(.h(
(19)
As is shown in Figure 4(b), we test the dependency contribution score i aa of each attention head in BERT for both wrong semantic dependency between "sign" and "?" and correct semantic dependency between "anthem" to "marry" in corresponding QA instance. In this case can observe there a group of attention heads (highlighted with bright color in the contribution heatmap) mutually contribute to the semantic dependency. We can also find the head group responsible for false dependency is clearly brighter than correct dependency, showing different pattern.
Extra Experiments and Results Firstly, we calculated the average number of top 5% contributing attention heads per layer for both correct and false semantic dependency across all BERT's failed QA cases (shown Figure 8). The results reveal the heads responsible for false semantic dependeney primarily distributed in later layers, whereas those contributing to correct semantic dependency are distributed across both earlier and later layers.
Secondly, We have analyzed the frequency of each top 5% contributing attention head for correct and false semantic dependency across all failed QA cases (a part of the result is shown in the main text in Figure 5). Here, we include all fine-tuned BERT models commonly used in QA tasks with high accuracy (F1 > 0.8) in Figure 9.
Additionally, We also calculated the average number of top 5% contributing attention heads per Layer for semantic dependency across successful QA cases in BERT series (the same number of failed QA cases randomly sampled from the SQuAD dataset) in Figure 10. The results show that when the model performs correctly, the heads responsible for the correct semantic dependency are mostly distributed in later layers compared to correct semantic dependency when the model fails QA task. Based on Figure& 8 and Figure 10, we surmise that The model tends to rely more on the attention heads in the later layers when answering questions, regardless of whether the answer is correct or incorrect.
Furthermore, we also count the frequency of each top 5% contributing attention head across successful QA cases (the same number of failed QA cases randomly sampled from the SQuAD dataset) Figure 11. We also found similar group of top attention heads responsible for correct semantic dependency when models fail or succeed in QA tasks, which means they QA spec-spe heads. The high overlap of the task-specific attention heads also shows that the same parameters contribute to encoding correct semantic dependency regardless of whether models succeed or fail in QA cases. Based on Figure 9 and Figure 11, we can conclude that such model mistakes in QA tasks can not be corrected by directly adjusting parameters such as simply pruning attention heads because correct and incorrect dependencies are controlled by the same group of parameters. We need other strategies to address the challenge of removing false dependencies without affecting the correct ones.
Discussion As highlighted in the main text, the attention mechanism plays a key role in propagating semantic information between tokens, ultimately enabling final layer tokens to encode semantic dependencies. In this paper, we focus primarily on analyzing the contribution of attention head parameters. Additionally, we notice that MLP layers may amplify irrelevant or erroneous semantics (Geva et al., 2023). In future work, we aim to extend our analysis to quantify the contribution of MLP layers semantic dependency.
21

---

A Lens into Interpretable Transformer Mistakes via Semantic Dependency
Average Number of Headls
Average Number of Heads
Average Number of Heads
Auerage Number Heads
3.0
0.5
0.0
0.0
1.5
1.0
0.
(a) BERT
(b) RoBERTa
(c) TinyRoBERTa
(d) ALBERT
Average Number of Heads
Average Number of teeads
Aversqe Number Heads
1.0
Average Number of Heads
0.5
0.0
10
20
Layer
Layer
Layer
Layer
(c) DistilBERT
(f) DeBERTa
(g) MobileBERT
(h) MiniLM
Figure 8: Distribution of top 5% contributing attention heads for correct (green) and false (red) semantic dependency across
all failed QA cases.
07
Requercy
6.2
Normal Reruercc
Normaloed Preueeeeeeee
Heus
commateee
Normatonn
0.1
14
10
15
20
..0
36
0.0
0
0.0
30
0.0
Layer
Layer
Ticmmlized inooueery
05
Nomaaltzed Frequeecy
Tv Freeieency
Norm feed Freouveryy
C S.
0.5
0
Head
Head
2
10
10
10
0.0
(a) BERT
(b) RoBERTa
(c) TinyRoBERTa
(d) ALBERT
0,6
Normaluee reequnccy
Nomalled Frequticc
Temoloed Peepeesc
0.4
Head
D.
Head
20
0.0
Layer
Layer
Layer
Layer
0.4
Ser setteeteee Inquuerty
Norattoo Preaverry
2
foooeed heatterry
Hard
Head
nog
0.2
26
10
20
10
6.
(c) DistilBERT
(f) DcBERTa
MobileBERT
(h) MiniLM
Figure 9: Frequency of each top 5% contributing attention head for correct (green) and false (red) semantic dependency across all failed QA cases.
22

---

A Lens into Interpretable Transformer Mistakes via Semantic Dependency
Average Number of Heaals
Auerage Number Heads
Average Number of Heads
Auerage Number of Heads
3.0
0.0
0.
(a) BERT
(b) RoBERTa
(c) TinyRoBERTa
(d) ALBERT
Average Number of Heads
Averaçe Number of teeads
Averate Number of Heads
Average Number of Hearss
0.0
10
20
Layer
Layer
Layer
Layer
(c) DistilBERT
(f) DeBERTa
(g)
MobileBERT
(h) MiniLM
Figure 10: Distribution of top 5% contributing attention heads for correct semantic dependency in failed QA cases (green) compared to successful QA cases (blue).
0.8
67
Noraateed Peeuuee Requeticy
05
Normaloon Fwwdueeeeeeeeeee Requercy
0
mommafidedd Requiee
0.2
0
Noraltonnd
14
10
15
30
0.0
36
0.0
0
0.0
30
0.0
Layer
Layer
Fresteency
Nomaa ited Freauercy
Cemalized Prreurenryy
tred Frequrecyy
Head
Head
30
10
0.
(a) BERT
(b) RoBERTa
(c) TinyRoBERTa
(d) ALBERT
Northal und Erequaiccy
Do
Non Natr Prosucciyy
C Nomatted (epe
Hoad
10
20
0.0
Layer
Layer
Layer
Layer
0.
0.4
Nat Inequertyy
NM nolted Frealisryy
2
Hard
Head
Head
Ra
20
10
(c) DistilBERT
(f) DeBERTa
MobileBERT
(h) MiniL.
Figure 11; Frequency of each top 5% contributing attention head for correct semantic dependency in failed QA cases (green) compared to successful QA cases (blue).
23

---

A Lens into Interpretable Transformer Mistakes via Semantic Dependency
A.5. Pesudocode for Section 5
Algorithm Evaluation of Semantic Dependencies
Input: Dataset with M instances, transformer model fo, number of perturbations K.
Output: The percentage that Anmuia is greater than A A Aseaai given the question and answer pairs where the model
makes mistakes.
Initialize count for each incorrect QA instance to H do No
Extract question tokens Q {q. J i=1 correct answer tokens Acomrct
{a() and incorrect answer tokens
Awrong {a I=
for each answer token al E Acorreat U Awrong do
for to do
if h: 1 then
2 a
else
z + RandomToken())
end if
Construct perturbed sequence z0(k) by replacing a& with z
Compute final layer representations (k) (z0tk))
end for
Compute original final layer representations L(org) f- (o
for each token j = 1 to N do
Calculate A 1241 end for
- K
2
[(oz))
2
Determine maximum dependency score for a. ووله max No end for
Determine maximum dependency score for correct answers: A A 14 max
Ne A'
k= 2
Determine maximum dependency score for wrong answers: A AAamngg QQ Axnme A Aumaa then
max Nw
A'
k=1
if A
count &- courd + end if
end for
Calculate percentage: p(fe) Return: p(fo)
24

---

A Lens into Interpretable Transformer Mistakes via Semantic Dependency
A.6. Symbol List
Table 11: Symbols and Their Explanations
Symbol
Explanation
z
The embedding of the i-th token in the (-th layer.
z
The embedding of the j-th token in the 1-th layer.
z;
(urg)
The original embedding of the i-th token in the (-th layer.
2
[()
The k-th perturbed embedding of the i-th token in the I-th layer.
A 12/
Semantic dependency score, which measures how the perturbation of token i at layer O affects token j at the final layer I..
N
The number of tokens in a token sequence.
K
The i-th token in layer O is perturbed K times to calculate the average change of the i-th token in layer L. K = 5 in our experiments.
M
The number of total perturbed token cases across all sequences we evaluate.
P(fo)
Percentage P of the cases that the transformerbbase language model fo matches our finding.
Ww
True semantically dependent word group for the i-th word in layer 0 based semantic dependency parsing.
Gzg
Truthful semantically dependent token group for the i-th token in layer 0 based on semantic dependency parsing.
Estimated semantically dependent token group for the i-th token using token perturbation.
Kwp
The number of top tokens most sensitive the perturbation of the input token. Kup is set to the size of Go In the experiment, we evaluate the overlap of Gog and top 5 tokens when the size is under 5.
S
Alignment score between the truthful (G,g) and estimated (G 2g) semantically dependent token groups
Estimated semantically dependent token group for the i-th token corresponding to token sequences and 82.
Gleft x!
GRighh
Estimated semantically dependent token group for the i-th token corresponding to concatenated sequences (82, and (#1, 82).
DAS(.)
Dependency Alteration Score, measuring the impact of irrelevant context sequence order changes on semantic dependencies in sequence.
L
The number of chosen semantically dependent tokens in the original token
sequence g0(s)), e.g., L = 5 when choosing the top 5 semantically dependent tokens for evaluation.
q!
The embedding of the i-th question token in the I-th layer.
a/
The embedding of the j-th answer token in the 1-th layer.
Aafl {
Semantic dependency score in QA task, which measures how the perturbation of i-th answer token at layer 0 affects j-th question token at the final layer L.
A'
Highest semantic dependency score above all semantic dependency between all question tokens and i-th answer tokens in a QA task.
A
A'seselia Annaii
Highest semantic dependency score above all semantic dependency between question tokens and answer tokens (correct wrong) in QA task.
25


Reviews:
[{"review_id": "WeFSDVnLQk", "reviewer": "Reviewer_gFV2", "summary": "The manuscript explores how transformer-based language models encode semantic dependencies and how semantic dependencies contribute to errors in model outputs. The authors propose a perturbation-based interpretability method to measure semantic dependencies. They mainly examined how changes in input tokens affect token representations within the models. There are mainly four discoveries: \n\n1. Tokens mostly retain original semantic information through transformer layers.\n\n2. Models generally encode truthful semantic dependencies in their final layers.\n\n3. Model mistakes frequently arise due to falsely encoded semantic dependencies\n\n4. Correcting these mistakes via direct parameter adjustments is challenging because the same parameters encode both correct and incorrect semantic dependencies.\n\nThey validate these findings with extensive experiments across various transformer architectures including BERT variants, GPT-2, and LLaMA 3, primarily using perturbation-based techniques on texts of diverse sources such as GSM8K, Yelp, and SQuAD.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. (Claim 1) Analysis of differences between GPT-2 (75%) and BERT (98.8%).**\n\n**A1.**  Thank you for your insightful finding about GPT-2.  \n\nWe believe that this is likely related to model complexity. Following your findings, we conducted additional experiments and calculated the percentages for GPT-2, GPT-2-Large, and GPT-2-XL across different datasets. The results show that larger GPT models achieve similar percentages to BERT, e.g., around 98%. Due to the character limit of the rebuttal, the table will be included later.\n\n---\n\n**Q2. (Claim 2) Including multiple automatic semantic dependency parsing methods**\n\n**A2.** Thank you for the great suggestions. We have followed your comments and included results using different semantic dependency parsing methods. Specifically, we used two popular dependency parsing methods: Stanza (Stanford NLP) and AllenNLP. The results for verifying truthful semantic dependencies encoded in the final layers are similar to those obtained with SpaCy. The table will be included later.\n\n---\n\n**Q3. (Claim 2) Cases where words span multiple subword tokens are not considered, requiring justification.**\n\n**A3.** Thanks for the comment.   Although model-estimated semantic dependencies can be easily obtained, the main challenge is that existing semantic dependency parser methods usually cannot measure dependencies at the subword level. This makes direct comparison difficult. To address this issue, we may need to manually annotate the semantic dependencies and compare them with those estimated by the models, which is costly and hard to scale. We have carefully acknowledged this in our main paper.\n\n---\n\n**Q4. (Claim 3) How the F1<0.6 threshold for errors is selected.** \n\n**A4.** The threshold of F1 < 0.6 for identifying incorrect answers was determined empirically. Since our goal is to assess whether incorrect answers are associated with incorrect semantic dependencies, an F1 score below 0.6 indicates that over 60% of the tokens predicted by the model differ from those in the original answer—strongly suggesting the answer is likely incorrect.\n\nTo further strengthen this analysis, and following existing work, we conducted additional experiments using more advanced ChatGPT models to compare the model’s answer with the ground truth and find incorrect cases. The results are similar with using F1<0.6.\n\n---\n\n**Q5. (Claim 3) how GPT-2 resulted in an F1 as low as 0.78%, some qualitative examples would help.**\n\n**A5.** Note that to ensure a fair comparison, we evaluated LLaMA and GPT models using the same zero-shot (0-shot) setting as BERT. This is the reason that they have a low accuracy. We conducted extra experiments using a one-shot setting, which aligns with official benchmark evaluations.\n\nBelow are two randomly selected qualitative examples of GPT2’s performance on QA tasks.\n\n- Question: Who was the Super Bowl 50 MVP?\n- Context: The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five solo tackles, 2½ sacks, and two forced fumbles.\n- Expected Answer: Von Miller\n- GPT-2 Answer: Peyton Manning\n- Analysis: Peyton Manning is not in the context. The model incorrectly associates Super Bowl 50 MVP with Peyton Manning instead of Von Miller.\n- Question: How many times was Cam Newton sacked?\n- Context: (Same as above)\n- Expected Answer: Seven\n- GPT-2 Answer: Cam was sacked three times\n- Analysis: The model misinterprets the numerical information.\n\n---\n\n**Q6. (Q2 in review) I'm wondering whether using controlled perturbations (e.g., synonyms) instead of random vocabulary could help strengthen validity.** \n\n**A6.** Thank you for this insightful question. To make it clearer, we have added more justification for using random vocabulary in our revised version. For both claim 2 and claim 3, we need to understand how semantic dependency is encoded in the final layer by replacing an input token *A* and observing which final-layer token changes (e.g., at position *B*).\n\n- Suppose that *A* and *B* have a strong dependence, that is, the semantic information of *A* is encoded in the final-layer representation at position *B*. \n- What we want to observe is that when *A* is replaced with another token, *B* changes significantly. This would suggest that *B* encodes information from *A*, indicating a semantic dependency between them.\n- However, if we replace *A* with a synonym (*A’*), the overall semantic meaning of the sentence may remain largely unchanged, and the model may treat *A* and *A’* similarly. \n- In this case, we may not observe a large change at *B*, making it difficult to conclude whether *B* was originally dependent on *A*, even if a true dependency existed. Therefore, we use random tokens to encourage semantic independence.", "claims_and_evidence": "The key claims in the paper are supported by extensive experiments, although some limitations exist.\n\n## **Claim 1**\nTokens mostly retain original semantic information through transformer layers. This claim is supported by high retention percentages in Table 1, though a deeper analysis and discussion on the differences between GPT-2’s 75% vs. BERT’s 98.8% would better support this claim. Overall, the claim is supported by the empirical evidence.\n\n## **Claim 2**\nModels generally encode truthful semantic dependencies in their final layers. High alignment scores in Table 2 validate this claim. One thing that might need attention is that SpaCy labels instead of human expert annotations are used as ground truth, which might incur bias inherent to SpaCy's models. Alternatively, I would suggest either including multiple automatic semantic dependency parsing, or manually annotate a small subset of the dataset to verify the consistency and strengthen the conclusion. Also, the case when a word spans multiple subword tokens is not considered, which might need some brief additional justification given that is a common scenario. \n\n## **Claim 3**\nModel mistakes frequently arise due to falsely encoded semantic dependencies. The claim is supported by QA task results (Table 3). I'm a bit confused how the F1<0.6 threshold for errors is selected, and despite the generative nature, how GPT-2 resulted in an F1 as low as 0.78%, some qualitative examples would help. \n\n## **Claim 4**\nCorrecting the mistakes via direct parameter adjustments is challenging because the same parameters encode both correct and incorrect semantic dependencies. Visualization (Figure 5 and Appendix A.4) shows that certain attention heads encode both correct and incorrect dependencies, and directly disabling these heads, such as head pruning, might hurt overall performance. However, I believe a word other than \"adjustment\" should be used to make the claim more accurate", "methods_and_evaluation_criteria": "## **Methods**\nThe authors mainly used the perturbation-based method to trace the semantic flow in transformers. One minor suggestion is that random vocabulary replacement might introduce noise as it might drastically change the sentence into nonsensical texts. I'm wondering whether using controlled perturbations (e.g., synonyms) instead of random vocabulary could help strengthen validity.\n\n## **Evaluation Criteria**\nThe authors tested their claims against a battery of datasets, which spans multiple domains (math, reviews, wiki, etc.) and tasks (QA, classification, etc.). The datasets used in this paper are adequate for the authors' purpose and are representative of broader research interest. The evaluation metrics used (both newly defined and existing) are reasonable and make convincing support for the authors' claims.", "theoretical_claims": "No formal theoretical proofs, but the math is overall sound. The link between parameter localization and dependency encoding could be more rigorously established as it is not very straightforward.", "experimental_designs_or_analyses": "Experiment designs are overall sound and empirical evidence is rather thorough. Optionally I would encourage: \n* Validating SpaCy-derived dependencies against human annotations, or cross validate with other tools.\n* Testing perturbation with synonym tokens (vs. random).", "relation_to_broader_scientific_literature": "Connects well with semantic dependency parsing, probing studies, and interpretability of attention mechanisms. Differentiates by focusing on token-level error analysis.", "essential_references_not_discussed": "Not that I am aware of. The detailed related works in A.1 are rather comprehensive.", "other_strengths_and_weaknesses": "## **Strengths**\n* The paper is well written and easy to follow. The details are clearly explained.\n* The core method is novel, simple and effective, and can be helpful for other related probing use cases.", "other_comments_or_suggestions": "Typos: \n\nFigure 4b: marry -> Mary; samantic -> semantic\n\nL248: Spacy -> SpaCy", "questions_for_authors": "**Q1:** Why was F1<0.6 chosen to define incorrect QA answers? How would results change with stricter/lenient thresholds?\n\n**Q2:** In your discussion section you mentioned \"For instance, replacing a token with a semantically similar yet different token may lead to significant variation depending on the model’s interpretation\", can you elaborate why this will be an issue and why random sampling from the vocabulary is better?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WbN8Yna4VY", "reviewer": "Reviewer_LWrx", "summary": "The authors investigate a way to measure token dependency and how varying levels of dependency affect transformer model performance, contribute to incorrect information, and encode semantic dependencies. Analyzing BERT, LLaMA, and GPT, they find that most tokens retain their original semantic meaning, with final-layer tokens usually encoding truthful dependencies, though they remain sensitive to context changes and order. They further find that errors arise when certain tokens falsely encode semantic dependencies, and understanding these mechanisms is hypothesized to enhance transformer model accuracy.\n\n## update after rebuttal\nI have revised my overall score to 4 (accept) as the paper makes a relevant contribution to structured explainability and presents solid empirical results. I think the paper misses some key prior works in detecting token interactions as summarized below. Assumign these minor changes to be implemented until the camera-ready version, I would vote for accepting the paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. (Claim 1) How much do these results depend on the specific run of sampling random tokens?**\n\n**A1.** Thank you for this important question. To ensure the stability of our results, we performed 5 independent random sampling trials for each token in the sequence and reported the average score across these trials. This ensures that our findings are not overly sensitive to a specific random sample.\n\nFollowing your suggestion, we conducted additional experiments using 10 independent random samples per token. The results remained very similar, further validating the stability of our results.\n\n---\n\n**Q2.  (Claim 1 and Methods)  The choice of the L2 distance but not other distance functions, e.g. dot products, cosine similarities.**\n\n**A2.** Thank you for this insightful comment! Following your comments.\n\nWe followed your comment and carefully added a discussion on our assumption for using  L2 distance in our revised version, including examples. Thank you for helping us further improve our paper.\n\nWe chose L2 distance based on the assumption that both magnitude and direction of the representation vector contribute to changes in semantic meaning. In contrast:\n\n- Cosine similarity captures only directional changes, completely ignoring magnitude.\n- Dot product can reflect magnitude to some extent but is heavily dependent on direction\n\nTo illustrate this, we will provide illustrative examples later due to limited space.\n\n---\n\n**Q3.  (Claim 1)  How scores of validation 2 were calculated.**\n\n**A3.** Intuitively, we computed the Validation 2 scores by modifying one input token at a time and measuring the percentage of tokens in the final layer whose representations change. We found that nearly all final-layer tokens are affected, suggesting that information from a single input token is distributed, though to varying degrees, across the entire final layer.\n\nIt’s worth emphasizing that this is not a major finding of our paper, therefore, we presented the results in the appendix. The only reason we highlight it in a few words in the main text is to draw attention to a key contrast with human language processing—humans seem do not propagate the information of a word to all others.\n\nWe do not yet know whether this behavior is beneficial or harmful for model performance, but we believe it warrants attention and further investigation. \n\n---\n\n**Q4. (Methods) Random token selection may introduce out-of-domain predictions, which leads to unreliable results.**\n\n**A4.** Thank you for this very insightful point! We have added your comment to Discussion section of our revised version.\n\n We believe measuring sensitivity or semantic relevance through gradients could provide valueble insights and are exciting directions for future work.\n\n---\n\n**Q5. (Essential References, Weaknesses)** The work overall does a good job at contextualizing and motivating the results. It lacks some related works from the interpretability community.\n\n**A5.** Thank you very much for your insightful comments regarding related work. Incorporating these references can significantly strengthen our paper by providing a more comprehensive overview of the interpretability literature and making the work more self-contained.\n\nFollowing your suggestion, we have revised the paper and carefully discussed the mentioned references in Appendix A1.\n\nAdditionally, we highlight key differences between our approach and existing lines of work:\n\n- **Feature attribution methods** primarily aim to assess the importance of individual tokens or features to the model’s output.\n- **Semantic dependency methods based on feature/token interactions** focus on study the contribution of combinations of features or tokens to model predictions.\n- **Feature binding methods** often do not test whether the model’s most confident output reflects encoded semantic dependencies, rather, many assume this relationship holds and to study downstream properties. In contrast, our method is designed to explicitly test the assumption by evaluating whether there is a dependence between the model’s output and the semantic dependency encoded in the final-layer token.\n\n---\n\n**Q6. (Weakness) Lack of theoretical contributions. Is feature dependency related to the magnitude of the attention score? How strongly does the dependency at a given layer correspond to the strength of the attention score?**\n\n**A6.** Thank you for the thoughtful comment. \n\nYes, we believe that feature dependency is directly related to the magnitude of the attention score. A mathematical formulation is provided in Appendix A.4.\n\nHowever, deriving a precise theoretical relationship under realistic assumptions between attention score and semantic dependency is very challenging due to the nonlinear and complex structure of neural networks. We acknowledge this limitation in our revised version and recognize it as an important direction for future research.", "claims_and_evidence": "- Claim 1: Most tokens primarily retain their original semantic information, even as they pass through the layers of transformers.\n    - Evidence:  Table 1 presents the fraction of tokens that retain their original information.  The score defined in Eq. 6 defines retaining original information as resulting in maximal change for token j when perturbing i.\n    - Reviewer Evaluation: The evaluating seems suitable to assess this claim, it remains unclear how much these results depend on the specific run of sampling random tokens. Also the choice of the L2 distance to assess semantic dependency is not clearly motivated, other distance functions may have been used, e.g. dot products, cosine similarities, etc. It was not entirely clear to me how scores of validation 2 were calculated. \n\n- Claim 2: A token in the final layer usually encodes truthful semantic dependency.\n    - Evidence: By using dependency trees the agreement between these and dependency scores is computed (Table 2).\n    - Reviewer Evaluation: This appears to be a good evaluation of the claim (given the limitations raised of the dependency scores raised in Claim 1) and supports the evaluation.\n\n- Claim 3: Model mistakes are linked to certain tokens that incorrectly encode information that is not semantically dependent.\n    - Evidence: Table 3 presents the fraction of failed answers on the SQuAD Q&A dataset when assessing the dependency strength     -between the answer and question token. It creates an empirical connection between wrong answer tokens having a stronger effect on the question token than the correct answer token.\n    - Reviewer Evaluation: This appears to be a simple yet sound evaluation approach.", "methods_and_evaluation_criteria": "- The methods are clearly defined and appear to be reproducible with reasonable efforts. \n- The method may suffer from some limitations (see below) that are not sufficiently discussed.\n- The selection of randomly selecting tokens may introduce out of domain predictions that can result in unreliable results. Alternatives would be to measure sensitivity or relevance via gradients/feature attribution.\n- The choice of the L2 distance to assess semantic dependency is not clearly motivated.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "- The experimental design is exhaustive and covers a representative set of models/architectures and tasks.\n- Analyses appear sound and appear reproducible with reasonable efforts.", "relation_to_broader_scientific_literature": "The work overall does a good job at contextualizing and motivating the results. It lacks some related works for the interpretability community that has come up with a variety of methods and approaches to investigate feature importance and feature interactions (see below), which should be added to the final manuscript.", "essential_references_not_discussed": "- Feature attribution to assess importance of tokens (in the context of Transformers and NLP)\n\n    - [1] Abnar, S., & Zuidema, W. (2020). “Quantifying attention flow in transformers”. ACL 2020\n\n    - [2] Ali, A., Schnake, T., Eberle, O., Montavon, G., Müller, K. R., & Wolf, L. (2022, June). XAI for transformers: Better explanations through conservative propagation. In International conference on machine learning (pp. 435-451). PMLR.\n\n- Semantic dependencies as interactions between features/tokens\n\n    - [3] Eberle, O., Büttner, J., Kräutli, F., Müller, K. R., Valleriani, M., & Montavon, G. (2020). “Building and interpreting deep similarity models”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3), 1149-1161.\n\n    - [4] J. D. Janizek, P. Sturmfels and S. Lee. “Explaining explanations: Axiomatic feature interactions for deep networks”, CoRR, 2020.\n\n    - [5] Schnake, T., Eberle, O., Lederer, J., Nakajima, S., Schütt, K. T., Müller, K. R., & Montavon, G. (2021). “Higher-order explanations of graph neural networks via relevant walks”. IEEE transactions on pattern analysis and machine intelligence, 44(11), 7581-7596.\n\n\n- Feature binding in language\n\n    - [6] Vasileiou, A. and  Eberle, O.. “Explaining Text Similarity in Transformer Models”, NAACL 2024.\n    - [7] Feng J. and Steinhardt, J.  “How do Language Models Bind Entities in Context?”, ICLR 2024.", "other_strengths_and_weaknesses": "Strengths: \n- The provided illustrations are well done and helpful to understand the approach.\n- The paper is overall very well written and clearly structured.\n- The discussion of the results was good and provided additional depth.\n\n\nWeaknesses: \n- Lack of discussing related approaches in the interpretability and explainable AI community.\n- Lack of theoretical contributions to better understand what drives feature dependency, i.e. is it directly related to the magnitude of the attention score?", "other_comments_or_suggestions": "- Caption of Figure 1 should more clearly state what validation 1 and 2 are.\n- Overall, I think it is a good paper that investigates clearly defined claims from an empirical perspective. The novelty is moderate and focused on language experiments.", "questions_for_authors": "- How strongly does the dependency at a given layer correspond to the strength of the attention score?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "9IbDxaEEiK", "reviewer": "Reviewer_VaRV", "summary": "This paper studies how semantic dependency changes within the model architecture by investgating the toekn values. Through experientment, the author find that: 1) most token retain original information as layer goes deeper. 2) truthful semantic information is encoded in the token in final layer. 3) wrong output is related to incorrect semantic dependencies. And the author finds that wrong and correct semantic information is encoded in same parameter, which causes difficulties to remove incorrect semantics.\n\n## update after rebuttal\n\nAfter rebuttal I opt to update the score to 3 since the responses have addressed my most concerns.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. (Claims, Chapter 3) Because of resnet, it's reasonable that the i-th token is the most sensitive one in final layer to change of the i-th token, which is not enough to draw the conclusion that most token retain their original semantic information.**\n\n**A1.** Thank you for this insightful observation regarding the influence of residual connections. We are not sure if we have interpreted your comment correctly, please kindly allow us to elaborate on our main point.\n\n- Our central claim is that semantic information encoded in a final-layer token can be observed by measuring its sensitivity to changes in the corresponding input token. This claim is independent of any specific architectural feature, such as residual connections.\n- Your observation is highly insightful. Residual shortcuts could encourage final-layer tokens to retain their original semantic information. For example, in a very simple 1-layer model with a direct shortcut from input to output, it is highly likely that the final-layer token closely reflects the original input token.\n\nAdditionally, we conducted additional experiments on different GPT model sizes. \n\n- Interestingly, although all models share the same residual architecture, larger models (e.g., GPT-2-XL) exhibit significantly stronger semantic retention at the final-layer token level than smaller models (e.g., GPT-2).\n- This is somewhat counterintuitive. One might expect that simpler models, like GPT-2, would retain more of the original token content due to fewer layers and less transformation. However, the results indicate oppsitely.\n- This suggests that semantic retention is also influenced by other factors such as model complexity. \n\n\n---\n\n**Q2. (Claims, Chapter 5.1) To make the analysis and result in 5.1 more convincing, it's helpful to provide a table similar to the confusion matrix.**\n\n**A2.**  Thank you for the insightful comment! To further make the analysis in Section 5.1 more comprehensive, we have followed your comment and included a table for each model. Due to space limitations, the tables will be attached later.\n\nThe results show that when the model correctly encodes the semantic dependency in the final-layer token, it usually provides the correct answer. Conversely, when the model produces an incorrect answer, the semantic dependency is often incorrectly encoded.\n\nThese findings highlight the importance of semantic dependency encoded in the final-layer token for model predictions.\n\n---\n\n**Q3. (Claims, Chapter 5.1) I think the percentage that the model makes mistakes when the maximum dependency score for incorrect answers exceeds that of correct answers, should also be calculated.** \n\n**A3.** The metric in our original paper explicitly follows your method. We have revised our paper by including your comments to make this point clearer.\n\n---\n\n**Q4. (Claims, chapter 5.1) The conclusion \"reducing the influence of false dependencies on outputs\" is unreasonable.**\n\n**A4.** We are sorry for this terrible confusion. \n\nWe intended to say that RoBERTa shows a lower proportion (69.20%) compared to TinyRoBERTa (77.94%), but a similar L1 score. This may indicate that RoBERTa has a more robust mechanism for reducing the influence of false dependencies on outputs.\n\nWe found your comment “a lower percentage means false dependencies ...”  is an important point, and we have carefully acknowledged it in the revised version.\n\nThank you again for your valuable feedback and for helping us improve the quality of the paper.\n\n---\n\n**Q5. (Methods) I think the selected perturbation token should not be semantically similar to the original token.** \n\n**A5.** Thank you for being so insightful. We follow the same idea that the perturbation token is randomly sampled from the full vocabulary of each model. The probability of selecting a semantically similar token is low. \n\n---\n\n**Q6. (Methods)  Recent open source models could be considered.**\n\n**A6.** The result will be provided shortly.\n\n---\n\n**Q7. (References) Semantic role labeling (SRL) advancements should be included.** \n\n**A7.** Thank you for your suggestion. We have included and added a discussion in Appendix A.1.\n\n- Semantic role labeling methods assign semantic roles to words in a sentence, which are similar to semantic dependency parsing methods.\n- Our method aims to explain how semantic dependencies are encoded in the final-layer tokens of transformers.\n- To evaluate whether transformer models encode truthful semantic dependencies in their final-layer tokens, semantic role labeling methods can be leveraged as a reference.\n\n---\n\n\n\n**Q8. (Questions) According to figure1 and validation2 in table1, is the causal mask removed in GPT-2 and LLaMA3?** \n\n**A8.** Thank you for your question. The causal mask is not removed. When an input layer token changes, final layer tokens in autoregressive models like GPT-2 and LLaMA3 exhibit zero change for tokens on its left (see Appendix A.3). We will emphasize this to avoid further confusion.", "claims_and_evidence": "chapter 3\n\nI think the experiment is not convincing enough. Because of resnet,  it's reasonable and apparent that the i-th token is the most sensitive one in final layer to change of the i-th token, which is not enough to draw the conclusion(most token retain their original semantic information).\n\nchapter 5.1\n\nThe conclution that \"higher percentages suggest that their architecture may be more susceptible to false dependencies when mistakes occur\" sholud be supported by more evidence. I think the percentage that model makes mistakes when maximum dependency score for incorrect answers exceeds that of correct answers,should also be calculated.\n\nThe conclusion that \"lower percentages, suggesting a potentially more robust mechanism for reducing the influence of false dependencies on outputs\" is wrong. Lower percentage means false dependency accounts for a small proportion in failed QA instances, which means there may be some other factors lead to wrong outputs. The conclusion \"reducing the influence of false dependencies on outputs\" is unreasonable.\n\nTo make the analysis and reasult in 5.1 more convincing, it's helpful to provide an table similar to the confusion matrix.", "methods_and_evaluation_criteria": "The paper mainly uses perturbation method to calculate semantic dependency between token in first layer and in final layer. I think the selected perturbation token should not be semantically similar to the original token. more detailed explanation to token perturbation (like the vocabulary) should be provided. \n\n The author compares different large models(Bert and GPT) to demonstrate the findings, but more recent open source models could also be considered. And it will be more objective to list parameter size in the table.", "theoretical_claims": "no theoretical claim and proof", "experimental_designs_or_analyses": "Discussed in *Claims and evidence, Methods and Evaluation Criteria.* section.", "relation_to_broader_scientific_literature": "The paper could strengthen its contribution by explicitly linking its findings to existing research on semantic dependencies and transformer models, highlighting how it advances or diverges from prior work.", "essential_references_not_discussed": "The key contribution of the paper is analyzing how semantic dependencies affect token behavior in transformer models, but it does not cite recent work on semantic role labeling (SRL) advancements, such as https://arxiv.org/abs/2502.08660, which introduces a novel method for capturing fine-grained semantic dependencies in transformers. This work could provide additional context for understanding how semantic dependencies are encoded and propagated across layers.", "other_strengths_and_weaknesses": "Discussed above.", "other_comments_or_suggestions": "A typo:\n\nIn equation(3), $FFN(z^l)$ should be $FFN(\\hat{z}^l)$", "questions_for_authors": "According to figure1 and validation2 in table1, is the causal mask removed in GPT-2 and LLaMA3?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E2WsjM3pWv", "reviewer": "Reviewer_1yCg", "summary": "The paper proposes a score for measuring \"semantic dependency\" of final token activations on various input tokens. Specifically, for a final layer token activation at token $j$, and a token $i$ in the same input sequence, the semantic dependency is the expected euclidean norm of the the change in final layer token activation at $j$ when $i$ is counterfactually perturbed to a random token.\n\nThe authors then measure various properties of their score, namely:\n- Showing that final layer activation is most dependent on its own token\n- Correlating their semantic dependency score with semantic dependency groupings from SpaCy\n- Correlating their semantic dependency score with failures to answer QA tasks\n\nLastly, the authors attributed how each attention head outputting at token $j$ contributes to semantic dependency when token $i$ is changed.\n\n## Update after rebuttal\nI do not think the rebuttal addressed the core issues that I raised in my initial review. For example, the authors did not make a convincing case that their experiments are in any way causal; to show causality one needs to causally intervene on semantic dependency. In addition, the authors did not provide any additional empirical evidence to address the limitations of the original experiments. I have therefore chosen to not increase my recommendation.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. (Claims) The authors use causal languageHowever, the experiments in section 5 are correlational.**\n\n**A1.**  Thank you for this insightful question. We believe this question stems from a misunderstanding of what we mean by \"cause\" in this context. We have revised the paper carefully to make this clearer. Please also kindly let us know of any part that may confuse.  Please kindly allow us to outline the logic of our reasoning:\n\n- **Causal claim:** Encoding incorrect semantic dependencies can lead to incorrect answers. This is a reasonable assumption even for humans. To answer questions correctly, we need to represent semantic dependencies accurately. \n- From the claim, we believe that, for neural networks, semantic dependencies must be somehow encoded in their internal token representations.  Based on this understanding,\n- In Section 4, we observe that in correct predictions, the final-layer token tends to encode semantically dependent words together.\n- In Section 5,  we observe that when semantically dependent words are not encoded together in the final-layer tokens, the model is more likely to make a mistake.\n\n\n---\n\n**Q2. (Experimental Designs) For Validation 2, I'd expect the vast majority of tokens to exhibit dependence.**\n\n**A2.** Thanks for the insightful questions. This is not a major finding, and we only briefly mention it in a few words in the main text, with the full results presented in the appendix.  Please kindly refer to A3 of our response to Reviewer LWrx for the motivation for highlighting this point. \n\n---\n\n**Q3. (Experimental Designs) I suggest using all two by two possibilities for Table 3.**\n\n**A3.** We have followed your insightful suggestion and included a table for each model. Due to space limitations, the tables will be attached in the comments.\n\n---\n\n**Q4. (Experimental Designs)** Why is the F1 score of llama-3 in table 3 so poor compared to all the other models? It is by far the best model in the table according to standard benchmarks such as MMLU.\n\n**A4.** To ensure a fair comparison, we evaluated LLaMA and GPT models using the same zero-shot (0-shot) setting as BERT.  We conducted an additional experiment using a one-shot (1-shot) setting, following the official evaluation method. The performance aligns with official evaluations.\n\n---\n\n**Q5.  (Broader Literature)  Looking only at the last layer representation ignores a lot of rich phenomena that occur within the language model.**\n\n**A5.** Thank you for the insightful question.\n\nPlease note that it is feasible to apply our methods to every layer of tokens. Our motivation for focusing on the final-layer tokens is that we aim to understand errors in the model’s output, and the final layer token should have the greatest influence on the output, as has also been acknowledged in existing works.\n\nWe also believe that looking at tokens from other layers can be important, and we would like to explore this further in future work.\n\n---\n\n**Q6.  (Broader Literature)  It is not clear to what extent the semantic dependency score says anything at all about the internal representations of the model**\n\n**A6.** The purpose of designing the semantic dependency score is to assess whether the model is more likely to make a mistake when semantically dependent words are not jointly encoded in the final-layer token. We believe this finding offers valuable insight into how internal representations relate to model errors, and it can help guide future method development. \n\n---\n\n**Q7.  (Broader Literature)  Consider an alternate score that uses KL in the log probs rather than the final layer activations.** \n\n**A7.** Thanks for the constructive feedback. \nWe believe the score you mentioned may be similar to the method proposed by Feng and Steinhardt (2024). \nHowever, their method fundamentally differs from ours in its assumptions and goals. \n\nIntuitively, they assume that the model’s most confident output reflects its encoded semantic dependency, and they use KL-based scores to study other downstream properties based on this assumption. \n\nNotably, their KL-based score cannot be used to test the validity of their assumption.\nIn contrast, our goal aligns more closely with testing that assumption, specifically, whether there is a statistical dependence between the model’s output and the semantic dependency encoded in the final-layer token. Note that these two lines of research can be complementary.\n\n---\n\n\n**Q8. (References) token-level ablations to causally identify dependencies are not engaged in the paper.  The semantic dependence is related to relational binding**\n\n**A8.** Thank you for this important point. In our work, we also employ token-level ablations (i.e., masking or replacing individual input tokens to observe changes in the model’s output). Please also kindly refer **A7** for relational binding. We will include these related works in the revised version.", "claims_and_evidence": "The authors use causal language like \"mistakes arise from the model’s tendency to encode false semantic dependency in tokens through transformer layers\", and \"model mistakes in QA tasks stem from incorrect semantic dependencies en-coded in question tokens\". However, the experiments in section 5 are correlational.", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "- Validation 2 in Table 1 is extremely sensitive. The quantity $\\Delta$ is an average of norms, which is guaranteed to be non-negative. The quantity measures the proportion of pairs of token for which the $\\Delta$ is positive, which doesn't seem very informative; by default I'd expect the vast majority of tokens to exhibit _some_ dependence.\n\n- The evaluation in Table 3 also is missing base rates. Specifically, the authors only showed that a large fraction of incorrect answers also have dependency scores higher on the wrong answer tokens compared to the correct answer tokens. However, this needs to be compared to the base rate of the latter happening. I suggest using something like spearman's rho, or even better, show the rates of all two by two possibilities.\n\n- Why is the F1 score of llama-3 in table 3 so poor compared to all the other models? It is by far the best model in the table according to standard benchmarks such as MMLU.", "relation_to_broader_scientific_literature": "Understanding how LMs represent information is an important area of study. However, looking only at the last layer representation ignores a lot of rich phenomena that occurs within the language model. Further, it is not clear to what extent the semantic dependency score says anything at all about the internal representations of the model; consider for instance an alternate score that is defined identically to the proposed score, but measures distances (say using KL) in the log probs rather than the final layer activations. These scores would be exceedingly similar, differing only by the choice of metric (since the logits are just a linear transformation of the scores), but the alternate metric is defined entirely behaviorally, rather than based on any internal representations.", "essential_references_not_discussed": "The idea of making token level ablations to causally identify dependencies has a long history that is not engaged with at all in the paper (Vig et al, 2020; Finlayson et al, 2021; Amini et al 2022).\n\nThe form of semantic dependency discussed in the paper is also related to the study of relational binding (Wattenberg et al, 2024; Feng et al, 2023)\n\nFinlayson, Matthew, et al. \"Causal analysis of syntactic agreement mechanisms in neural language models.\" arXiv preprint arXiv:2106.06087 (2021).\n\nVig, Jesse, et al. \"Investigating gender bias in language models using causal mediation analysis.\" Advances in neural information processing systems 33 (2020): 12388-12401.\n\nAmini, Afra, et al. \"Naturalistic causal probing for morpho-syntax.\" Transactions of the Association for Computational Linguistics 11 (2023): 384-403.\n\nWattenberg, Martin, and Fernanda B. Viégas. \"Relational composition in neural networks: A survey and call to action.\" arXiv preprint arXiv:2407.14662 (2024).\n\nFeng, Jiahai, and Jacob Steinhardt. \"How do language models bind entities in context?.\" arXiv preprint arXiv:2310.17191 (2023).", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "- typo in line 199, second column. repeated subscript $i$, one of them should be $j$.\n- same place: presumably, you meant $i \\le j$ for autoregressive models, and not for all $i, j$.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper examines how transformer models represent semantic dependencies in question answering. It presents four significant observations from the study, which most reviewers find valuable to the community.
The paper can be further improved by providing a more comprehensive overview of related works from the community. It would be great if the authors can discuss how the findings in this paper can benefit other practitioners in the community.
Decision: Accept (poster)