Dissecting Bias in LLMs: A Mechanistic Interpretability Perspective
Authors: Zubair Bashir
Bhavik Chandna
Procheta Sen
Venue: Accepted by TMLR ()

Abstract: Large Language Models (LLMs) are known to exhibit social, demographic, and gender biases, often as a consequence of the data on which they are trained. In this work, we adopt a mechanistic interpretability approach to analyze how such biases are structurally represented within models such as GPT-2 and Llama2. Focusing on demographic and gender biases, we explore different metrics to identify the internal edges responsible for biased behavior. We then assess the stability, localization, and generalizability of these components across dataset and linguistic variations. Through systematic ablations, we demonstrate that bias-related computations are highly localized, often concentrated in a small subset of layers. Moreover, the identified components change across fine-tuning settings, including those unrelated to bias. Finally, we show that removing these components not only reduces biased outputs but also affects other NLP tasks, such as named entity recognition and linguistic acceptability judgment because of the sharing of important components with these tasks.

Published in Transactions on Machine Learning Research (12/2025)
Dissecting Bias in LLMs: A Mechanistic Interpretability Perspective
Zubair Bashir* Indian Institute of Technology, Kharagpur
zubartasshirkkggiin..kkg..
Bhavik Chandna"
bchandna Ruccd edu
University of California San Diego
Procheta Sen University of Liverpool
procheta.sem@@iivrpooolll
Reviewed on OpenReview: https: // openreview. net/ forum? id= EpQ2CBJTyD
Abstract
Large Language Models (LLMs) are known to exhibit social, demographic, and gender biases, often as a consequence of their training data. In this work, we adopt a mechanistic interpretability approach to analyze how such biases are structurally represented within models such GPT-2 and Llama2. Focusing on demographic and gender biases, we explore different metrics identify the internal edges responsible for biased behavior. We then assess the stability, localization, and generalizability of these components across datasets and linguistic variations. Through systematic ablations, we demonstrate that bias-related effects are highly localized, often concentrated in a small subset of layers. Moreover, the identified components change across fine-tuning settings, including those unrelated to bias. Finally, we show that removing these components not only reduces biased outputs but also affects other NLP tasks, such as named entity recognition and linguistic acceptability judgment, because of the sharing of important components with these tasks. Our code is available at https://githut: http://aaaaaaa000///111 http://///gghhtooo/////aaaaooooa _Bas
1
Introduction
With the growing deployment of Large Language Models (LLMs) in high-impact domains such as education, healtheare, law, and content moderation, ensuring their responsible has become increasingly critical (Brhhane et al. (023 Among the many ethical and societal challenges associated with LLMs, bias remains one of the most pressing and pervasive concerns. Numerous studies have demonstrated that LLMs can reflect and even amplify harmful social, demographic, and gender biases in their outputs (Acerbi & Stubbersfield (2023)). These biases often manifest. in subtle yet consoquential forms, including stereotyping, uneven sentiment associations, and disproportionate representation. When integrated into downstream applications, such biases can result in unfair treatment, discrimination, or the spread of misinformation. Prior work on debiasing LLMs has predominantly explored approaches such as fine-tuning or data augmentation (Han et al. 2024): Gallegos al. 20224). A complementary line of research has investigated the role of individual neurons attention heads in encoding gender bias (Vig et al. (2020)). However, these studies often lack generalizability, as they tend to focus narrowly on a single type of bias and specific model architecture. Morcover, there remains limited exploration of key propertics of bias-related components, such as thoir localization, stability, and faithfulness altribution methods, and overlap with features relevant to other downstream tasks.
Equal contribution.

---

Published in Transactions on Machine Learning Research (12/2025)
Input
mi
m9
loots
Figure 1: Circuit Diagram for Positive Demographic Bias in a) GPT-2 Small, b) GPT-2 Large, and c) Llama-2. Green colour shows Input node, Orange colour shows Attention Head, Purple colour shows MLP layer, and Yellow color shows Logits node, The description of different types of nodes can be found Table 6 Appendix A. Circuit diagrams for gender bias are shown in Appendix A.9
To address the above-mentioned limitations, this work investigates demographic and gender bias in GPT-2 Small, GPT-2 Large, and Llama2, Qwen2-0.5b, Gemma-2-2b. In demographic bias, LLMs gencrally tend to predict mostly positive or negalive words for a particular nationality. For example given an input 'Afghan people are so', LLM completes it with negative words like 'Afghan people are so poor'. Similarly, gender bias tends to favour either the male or female gender for certain professions. An example of gender bias 'The woman worked nurse. The man worked software engineer'. It can be observed from the example that LLM assigned female gender to the 'nurse" profession and 'male' gender to the 'software engineer' profession.
In our research scope, we use Mechanistic Interpretability (MI) (Olah (022)) )related methodologies to identify important components within LLMs. Our central question is: Can demographic and gender biases be localized within distinct substructures of an LLM's architecture? If such biases can indeed be attributed to identifiable components, this opens the possibility of targeted interventions that is, mitigating harmful behavior by modifying or ablating responsible components, rather than relying on full-scale retraining or costly data augmentation. Out of different types of components (e.g., nodes and edges) available for analysis, specifically focus identifying the edges responsible for bias using Edge Attribution Patching (EAP) (Syed et al. (2024)) approach. An edge refers to connection between two computational nodes, such neurons, attention heads, or layer outputs. typically representing the flow of information between adjacent layers during the forward pass (Syed et al. (2024)). ). Figure shows a sample of example circuits (subgraph consisting of important edges [ for demographic bias in GPT-2 Small, GPT-2 Large and Llama-2. The key contributions of this work are as follows:
a) Localized Encoding of Demographic and Gender Bias in LLM Edges We conduct comprehensive analysis across GPT-2 Small, GPT-2 Large, and Llama-2 to investigate whether demographic and gender bias is encoded in localized manner. Our study employs different approaches to identify the edges most strongly associated with biased behavior. As illustrated in Figure 2 and 3 our study concludes that demographic and gender bias are mostly localized into certain odges and layers across different models.
b) Instability of Important Edges Across Lexical, Syntactic, and Fine- Tune-Tuning Variations We evaluate the generalizability and stability of the identified edges across multiple dimensions, including different types of bias (e.g,, gender, demographic), fine-tuning settings, and variations in grammatical structures. Our analysis (Figure and 5 concludes that the edges identified for biased behavior don't remain consistent under perturbations to both the model and input text space.
2

---

Published in Transactions on Machine Learning Research (12/2025)
<) Bias-Related Related Edge Overlap with Other Language Understanding Tasks We investigate the extent to which edges identified as important for biased behavior are also functionally involved in broader language understanding tasks. To probe this, we selectively corrupt edges associated with bias and measure the resulting impact on performance across a diverse set of unrelated NLP tasks. This analysis reveals the degree of functional entanglement between bias-related circuits and general linguistic competence, offering insights into whether bias can be mitigated without compromising core language capabilities.
The paper is organized as follows. In Scction 2
we describe existing works related to bias and mechanistic interpretability in LLMs. Section B describes the methodology used for identifying the important components, Section 4 describes the experiment setup, Section D describos results, and Section 0 concludes the paper.
2 Related Work
Bias in LLMs A growing body of research has investigated various forms of bias in LLMs. For instance, Kotek ct al. (023) highlighted profession-relatt related gender bias in GPT-3.5 and GPT-4, while Kamruzzaman et al. (2024) demonstrated that models such as GPT and Llama-2 exhibit systematic biases across underexplored dimensions, including age, physical appearance, academic affiliation, and nationality, using sentence completion tasks. Similarly, [Soundararajan & Delany (2024) observed gender bias in LLM-gencrated text. A broader overview of bias evaluation and mitigation strategies is provided in Navigli et al. (2023). The work in Bolukbasi 2016bb proposed debiasing approach for gender bias in word vectors.
et al.
MI aims to reverso-engineer trained neural networks, similar to the analysis of compiled software binaries Olah (022). The central goal is to identify and understand the role of internal components- as neurons, attention heads, and edges that give rise to specific model behaviors, including tasks like indirect object
identification or bias propagation Olah et all] (2020): Meng et al. (002)); Geiger et al. (2021); : Goh et al. (2021) ;
Wang et al. (2022). Recent work by CComyy et al. (2023) introduced an automated framework for discovering
task-relevant components; while Wu et al. (2023) proposed causal testing method to determine whether a
network implements specific algorithms. Nanda et al. (0023) leveraged circuit-based analysis to explain the
grokking phenomenon. In complementary work, Goldowsky-Dill et al. (2023) refined our understanding of
induction heads in LLMs, and Katz et al. 2024 employed MI techniques to trace information flow through transformer-based architectures.
MI for Bias Analysis Another thread of MI work has recently been harnessed to dissect and mitigate biases in LLMs. Vig et al. (2020) pioneered causal mediation analysis to pinpoint a small subset of neurons and attention heads in GPT-2 that drives gender bias. Chintam et al. (2023) combined automated circuit discovery and targeted ablations to surgically edit bias-carrying components, substantially reducing stereotype propagation with minimal impact on overall performance. The study in Kim et al. (2025) investigated how political bias or perspective is encoded in LLM internals. Kim et al. (2025) found that models learn a remarkably lincar representation of political idcology (c.g. liberal to conservative) within their activation space. However, prior work has not systematically examined the generalizability of bias-related components, nor explored key properties such as their localization, stability under perturbations, and overlap with components involved in broader language understanding tasks. To address these gaps, we conduct a comprehensive analysis of these properties across multiple model architectures.
3 Edge Attribution in LLMs
To investigate how bias is encoded within LLMs, we adopt a causal intervention framework assess the relative importance of individual edges within the model architecture. As described in Section [ an edge generally represents the flow of information between adjacent layers during the forward pass. Each edge is parameterized by a weight, which governs the strength of information transfer between the connected nodes.
A naive strategy would involve iteratively ablating individual edges and measuring their effect on model outputs. However, such exhaustive interventions are computationally expensive at the scale of modern LLMs. To address this, we leverage the Edge Attribution Patching (EAP) technique introduced by Syydd et al. (2024). which offers more efficient approximation of causal importance. EAP assigns an attribution score to each
3

---

Published in Transactions on Machine Learning Research (12/2025)
edge, reflecting its contribution to particular model behavior. Mathematically, EAP measures the value described in Equation for each edge.
[2Eeten | do(E Ecorr)) L(Xeteon))
(1)
In Equation | L is metric with respect to which measure the effectiveness of a task, and L( shows the value of metric when clean sample is provided to the network and [[Edtean do(E Ccorr)) represents the value of metric when corrupted activation (i.e. activation oblained from corrupted input (eorr) is applied to an edge for which we want to find the importance. For the rest of the edges, clean activation applied. If a set of edges is important for a particular task then providing corrupted values to those edges will reduce the value of [[Eetaa)) significantly. Attribution patching is an approach (Syed et al. (2024)) that used compute the value of Equation T with only two forward passes and one backward pass through the network. This makes EAP computationally efficient.
3.1 Bias Metric Computation (I)
There is currently universally accepted metric for quantifying demographic gender bias in LLMs. Similar to Qiu et al. (2023), we investigated two alternative formulations designed to capture different aspects of bias. In Equation m is the total number of samples on which L1 is computed.
m7
k
1 =
Σ
Ppos(1) - Σ Pneg(()) 5=1
(2)
Equation 2 shows one variation of metrie denoted L1- In Equation 2 we compute the difference in aggregate probabilities assigned to positive/male (i.e. Ppos) versus negative regative/femall female (i.e. Pneg) tokens among the top-k predicted next tokens. This captures the relative skew in the model's output distribution. Higher absolute value of L, will show higher bias in an LLM.
m
L
Σ j=1
Prss (i)
(3)
m
Equation 3 shows the second variation of the bias metric denoted as L2. Equation 3 isolates the cumulative probability assigned exclusively to positive /male tokens within the top-k predictions, offering unidirectional measure of bias. By comparing these two formulations (i.e., In and I2),, we aim to understand which metric more effectively identifies the edges responsible for encoding demographic and gender bias in EAP.
3.2 Defining Corrupted Samples
As shown in L'quation | generating appropriate corrupted samples is critical step in estimating the EAP score. To maintain consistency in attribution, each corrupted sample must preserve the token length of its corresponding clean imput. To achieve this, we adopt the Symmetric Token Replacement (STR) strategy proposed ZZhan & Nanda (2024), which ensures structural alignment between clean and perturbed inputs while introducing minimal semantic drift. For demographie bias, we explored two approaches: Ci, where the country name in the sample was replaced with an out-of- of-tiiiibbbttion token, such 'abe', and C2. where the country name was replaced with 'Emirati', the nationality for which we did not observe any demographic bias in LLM outputs. Similarly, for gender bias, we applied two strategies: C1, where gender-sensitive tokens like profession names (e.g,, doctor) were replaced with an out-of-disttibuttio word like 'xyz', and C2, where we used the gender-neutre term 'broadcaster', chosen because LLMs showed the most gender-neutral neutral predictions for this term. Table [ presents the structure of both the clean and corrupted samples used in our experiment. setup.
3.3 Localization of Bias
As described in Section I one of the objectives of our experiments is to understand whether demographic and gender bias is encoded within certain components in LLMs. We employ two principal strategies to assess
4

---

Published in Transactions on Machine Learning Research (12/2025)
Corrupted Sample
Clean Somple
C₁ people As expacted, Ale people are so age didd mistaker everyone sodeled The rye did a mitracl,, and everyone prateed
(DSS1) Afyhan people
Emárati
(DS52) As especiad, A Afphom people ane
As sspectod, Emiraali people are The Arsaddmnter rid mistale, and eorryon wookled The broadcaster did a miracle, and everyone praised
(GSS1) dottor
and everyone wcalded (G5S2) The doctor did a mincce, and everyone peatoed
Table 1: Variation of Clean and Corrupted Samples (l.e. C1. C2) Used for Different Blas Sotup. DSS stands for Demographle Sensitive Structure, GSS standa for Gender Sensitive Structure.
localization. Firstly, we evaluate the extent to which the highest-scoring edges (obtained from EAP method described in Section 3 contribute to the overall value of the bias metric (L) compared to basclinc model (i.e. where all edges are used). If bias is localized within specific edges, then ablating a greater number of important edges from the model architecture should lead to a more substantial decrease in the overall metric value for bias in the model. Additionally, we also analyze the layer-wise distribution of the important edges to identify the origin of the important edges within the model architecture.
3.4 Stability of Important Edges
Here, our primary goal to explore the stability of important edges identified using the EAP approach described in Section g By stability, we refer to how much the important edges change with respect different. criteria. We primarily focused on three different criteria. Each of them is described as follows.
C1: Stability with respect to Grammatical Structures Here we investigated generalizability across different grammatical structures (i.c. DSS1, DSS2, GSS1 and GSS2 described in Table D. Even if the grammatical structure changes, the gender/ /emder//emooraphiic bias present in a sentence is of the same nature. If the LLMs could generalize regarding bias, then the important structures related to bias remain the same for all the grammatical structures. To investigate this, we tested the different grammatical structures mentioned in Table I The difference between the grammatical structures for both demographic and gender bias that, in demographic bias, the semantic meanings of the two variations are almost similar. In contrast, for gender bias, the semantic meanings of the two sentences are different. To estimate the stability, we have primarily computed the overlap of top-K edges across different variations.
C2: Stability With Respect to Different Types of Bias Any kind of bias essentially means a preference for certain types of tokens over others. Hence, the motivation of this set of experiments was to check if there is any similarity between the important edges encoding different kinds of bias. In our rescarch scope, we looked into only demographic and gender bias.
C3: Robustness with respect to finetuning In this set of experiments, we investigate whether the important edges change concerning fine-tuning on different types of data. Here, we primarily did two different sets of experiments. Several studies Zmigrod et al. (2019); Han et al. (2024); Morerio et al. (2024) have demonstrated that targeted fine-tuning with augmented data sitta--ccciiiallly curated to address the bias category of interest can effectively mitigate biases in language models. In that direction, for the first set of experiments, we finetuned the model with a datasel where positive things were said about the countries countris/prrofessionn professions where there was negative demographic bias and vice versa. The objective of this experiment was to check whether debiasing the model by finc-tuning changes the important edges for generaling sentences of the same grammatical structure. In the second set of experiments, we finetuned the model on a dataset where there was no overlap with the bias-related sentences or test topics. The bias of the model doesn't 't change after this. We wanted to check whether the structures important for the bias also remain the same or not.
3.5 Leveraging Important Edges for Debiasing LLMs
Existing debiasing techniques predominantly focus on data augmentation, modifications to the training objective, or fine-tuning the model. While effective, these approaches inherently require retraining, which can be computationally expensive and infeasible in scenarios where access to model internals or training infrastructure is limited. To overcome this limitation, we propose a novel inference-time debiasing strategy that leverages the important edges identified via the EAP method (Section 3 Our approach involves
5

---

Published in Transactions on Machine Learning Research (12/2025)
substituting the activations (logits) along bias-associated edges with those obtained from a corrupted version of the input, while preserving clean activations along the remaining edges. This enables us to reduce biased behavior without altering the model parameters. Notably, this method requires no additional training or access to gradients and can be implemented entirely at inference time. To implement this technique, a corrupted variant of the original input is provided alongside the clean inpul. In our research scope, we automatically generated a corresponding corrupted input of the same length for each original input. The details of this setup are described in Appendix A.6
Prior work Kaneko et al. 2023 has demonstrated that debiasing can lead to performance degradation across various downstream tasks, suggesting that bias-associated components may also contribute to task-relevant behavior. To assess the broader impact of our debiasing method, we evaluated its effectiveness on a range of standard NLP tasks. Specifically, we apply the corrupted logits approach to multiple settings to investigate whether the mitigation of bias incurs a trade-off in task performance.
4 Experiment Setup
For demographic and gender bias-related experiments, we used two different types of datasets. The Demo- graphic Bias dataset used in our experiment is from an cxisting study in Narayanan Venkit et al. (2023) According to the findings in Narayanan Venkit et al. (223). GPT-2 and Llama-2 models exhibit nationality bias in text generation. This dataset was constructed by including the names of all 224 nationalities globally. The study in Narayanan Venkit et al. (2023) identified a negative bias towards certain nationalitics among these 224 nationalitics. Our rescarch specifically targets two distinct sentence structures (Described in Table D that mention nationality. We then assess the bias in GPT-2 or Llama-2 by examining the text completion of these sentence structures. The detailed descriptions about the model architecture of GPT-2 and Llama-2 models used in our experiment are given in Appendix A.2
To compute demographic bias in a text completion setup, we get the top-k next token predictions for every sentence (k- = 10 in our case, explained in detail in Appendix [A.I] by the model and for every sentence we concatenate cach of these predictions separately to the sentence and chock the sentiment scores of resulting sentences using Distilbert-base-unee uncased model HF Canonical Model Maintainers (2022). If the sentiment of the sentence is positive, then we assume that the token predicted by the model was positive and vice versa. For quantitatively computing the bias of the dataset 5 we used a metric (similar to L, in Equation 2 which computes the difference between positive and negative probabilities. Consequently, the Demographic Bias metric should be positive in the case of a Positive-Bias Dataset and negative in the case of a Negative- Bias Datasct. Using the methods described above, we divide our demographic datasct into two categorics: Positive- Biai datasel and Negative-Bias dataset. Tf the sum of the probabilities of next token predictions (for which the sentiment is positive) is greater than or equal to the sum of the probabilities of next token predictions (for which the sentiment is negative) for sentence, it is classified into the Dositivo-Bias datasct and vice versa.
To understand Gender Bias in models, we used the set of 320 professions chosen and annotated from Bolukbasi et al. (2016b). It is an exhaustive list of gender-specific and gender-neutral terms extracted from the w2vNEWS word embedding dataset Bolukbasi et al. (2016a). The dataset was formed on similar grounds to the Demographic one, using sentence structure prompts of mainly two types as described in Table I Using the gender dataset, we compute the bias in a similar way to demographic bias. The only difference is rather than using the sentiment score of sentence, compare the predicted token with the exhaustive set of male and female-specific common words mentioned in Bolukbasi et al. (2016b). Each predicted Remale-Stcrrotypicalfff
word is then assigned to three groups: Male-Stercottyica((MM5 (Gnder-Neuutra(GNN) Similar to demographic datasel, we divide our gender data into two categories: dataset. If the sum of MS probabilities (of next token predictions) is
and
Miae-B dataset and Female-Biass greater than or equal to the sum of FS probabilities (of next token prodictions) for a sentence, is classified into the Biased dataset and vice versa. Detailed dataset statistics for both demographic and gender bias and the corresponding bias estimations are given in Table 5 in Appendix A.T
6

---

Published in Transactions on Machine Learning Research (12/2025)
Change
Metric
Change in Metric
Ris Type
Configuration
GPT-2 Small
Llame-2
Qwan-2-0.Sh
Gemmaa
Biaw Type
GPT-2 Small
Liama-2
Qwwn-2-0.Sh
Gemmnn
Ia, Ci
0.2887
8:2218
0.2322
0.2128
G88
0.15e2
0.1831
0.2161
0.3678
DSS1pa
L, C2
0,2691
0.3441
0.0462
0.6021
GS5
0.1599
0.2261
0.1812
0.2168
DSS1pa
La. C
0,21111
0,2961
0,2081
0.3267
0.1498
0.2418
0.2315
0.2179
DS5
by Cs
0.1982
0.2007
0.0148
0.1052
G88
0.0259
0.2425
0,1521
0.2001
DS81ma
L. Ci
0,2887
0,2583
0,2233
0,2082
GSSlaer
0,2276
0.2939
0.2461
0.2718
DS8
0.1653
0,2804
0.3012
0.2189
GS8lacy
0.2688
0.2322
0.2128
DS91ar
La. C1
0,0653
0,0093
0.2351
0.3416
G9Slnee
0,1530
0.1865
0.2031
0.2541
DS8
0,0758
0-1520
0.1015
0,2011
GSS1are
0.2333
0.1511
0.3211
0.1301
Table 2: Variation in change in Metric value under different metric (l.e. L1 and La) and corruption configurations Cn and Cs. Each change value is normalized with respect to the original metric value produced by the model. For each category bias (i.e. DSS1pas DSSlneg. GSSIpar, GSSlneg) the lowest change in metric value is boldfaced.
Implementation Details We primarily experimented with the Hooked- Transformer from Transformer- Lens repository which offers a modular and transparent framework for studying the internal mechanisms of transformer models like GPT-2 and Llama-2. For the finetuning experiments, the pretrained GPT-2 and Llama-2 were finetuned to examine the change in underlying circuit. with respect to two different, types of datasets. In one variation, the model was tunettned on a Positive Dataset where all countries were given positive tokens (for gender bias case it is male gender bias) so that the finctuned model is biased toward positive sentiment irrespective of the nationality. The goal was to examine whether this fine-tuning approach alters the key edges responsible for generating sentences with similar grammatical structures. In the second variation, the model was fine-tuned on a Shakespeare Dataset, which consisted of Shakespeare text completely unrelated to the test sentences or topics. This experiment aimed to evaluate whether fine-tuning on entirely different content affects the edges contributing to bias. We conducted all the experiments in a computing machine having two A100 GPUs.
Based on the above discussion we would like to note that we will use the notations DSS1, DSS2 to describe demographic bias in the grammatical structures described in Table ! Similarly GSSI and GSS2 notations will be used to describe gender bias in the grammatical structures described in Table [ We will use the terminology DSS1 pos/neg (GSS1 pos/neg) to describe positive (male) or negalive (female) bias.
5 Results
As described in Section we show the effect of variations of corruption technique (described in Table and metric L (described in Section 3.1 in EAP for bias identification in Table 2 The 'Change in Metric" column in Table computes the difference in metric value when all the edges are used vs. only the important edges are used. From Table 2 it can be observed that the difference in metric value (i.e. change in Metric column in Table 2 cxhibits minimal change across variations in the corrupted structures (i.c. C1 and C2). However, when comparing L, and L2, it is evident that the absolute difference in metric values are mostly much smaller (except for DSS1 dataset in Llama-2) for Le than for L1. Smaller change implies that set of important edges identified by EAP performs almost similarly as the whole model where all the edges have been cosidered. Consequently, bias analysis experiments. We also show the top 3 edges corresponding to different types of bias across different types of models in Table 7 in Appendix A.4
used L2 and C2 as metric for all the remaining demographic and gender
Figure 2 and 3 show the results for localization bias-related components in GPT-2 and Llama-2. In Figure 2 we report, for each layer the LLM, the number of important edges- dess-lyssiinnnng only those layers that contain more than 20% of the total important edges to highlight the non-uniform distribution across the model architecture. We can observe from Figure 2 that for all the models (i.e. GPT-2 Small, GPT-2 Large and Llama-2) 2) only a few layers contribute to the important edges for bias (i.e. For GPT-2 Small it is layer 2 6, for GPT-2 Large it is primarily 9, 10, 20, 34,35 and for Llama-2 it is 0-11 and 30-31) 31). The observation from the Figure 2 confirms that bias is encoded in certain layers only. Since there is a significant overlap between important edges for DSS1 pos (GSS1pos) and DSSLneg (GSSIneg), in Figure 2 we only show the edge distribution for DSSL and GSS1 It can be observed from Figure 3 that within 40% of the top odges for GPT-2 Small and Llama-2 models, the motric value drops by more than 90%. For GPT-2 Large, it

---

Published in Transactions on Machine Learning Research (12/2025)
requires 60% of the edges to drop the metric value by 90%. Consequently, say that demographic and gender bias are encoded within few edges in GPT-2 Small, GPT-2 Large, and Llama-2.
50
20
Number
(a) GPT-2 Small DSS1
(b) GPT-2 Large DSS1
(c) Tlama-2 DSS1
50
ammportant
Layer
(d) CPT-2 Small on GSS1
GPT-2 Large on GSS1
(f) Llama-2 on GSS1
25-
uggn
Mypl
(g) Qwen2-0.5b onDSS1
(h) Qwen2-0.5b on GSS1
Gemma-2-22 on DSS1
(i) Gemma 2emmma-2-2b 2-2b on GSS1
Figure 2: Layerwise important edge distribution for demographic blas (DSS1) and Gender bias (GSS1) across different models (l.e. GPT-2 Small, GPT-2 Large, LLAMA-2, Qwen b and Gemma2-2b from left right).
Figure and illustrates the generalizability and robustness of the edges identified using the EAP approach. Figure 4 the three confusion matrices show the overlap among the top & edges for both demographic and gender bias. There is significant overlap between positive and negative biases of the same type (i.c., demographic or gender). However, there is minimal no overlap between demographic and gender biases. This pattern is consistent across GPT-2 (Small & Large), Llama-2, Qwen2-0.5b and Gemma2-2h models.

---

Published in Transactions on Machine Learning Research (12/2025)
win
(a) GPT-2 Small
(b) GPT-2 Large
(c)
Llama-2
Figure 3: Drop in value with % of Edge Ablation from GPT-2 Small, GPT-2 Large and Llama-2 different eon configurations (i.e. DSS1pori DSS2pox: DSSlnea- DSS2neg GSS1por GSS2 GSSInegi GSS2nea).
Lans
Dng p01s
Nac 5691
Preas 2011
the Swer
New
Ale
Sver
cars
Mic.
Prs legs
Naw ove
Figure 4: Plot showing overlap edges by EAP for GPT-2 Small, GPT-2 Large, Llama-2, Qwen2-0.5b and Gemma2-2b over different blas and sentence structure variation.
These results suggest that the similarity and dissimilarity patterns across different circuit variations are consistent across models. It also shown that there is very less overlap between the edges responsible for demographic and gender bias. An interesting observation from Figure that, for demographic bias under grammatical variation (i.c., DSS1 and DSS2), the sets of important edges exhibit minimal overlap. In contrast, gender bias under similar variations (i.e., GSSI and GSS2) reveals substantial degree of overlap, suggesting greater structural consistency in how gender-related information is represented within the model. We have shown similar analysis with another causal intervention technique Marks et al. (2025 in Appendix A.8
To evaluate the stability identified edges (i.e. described in Section edges in the pre-trained model and the fine-tuned model in Figure Fine-tuning was performed on two different datascts (i.c. Positive and Shakespeare as described Section Interestingly, as shown in the figure, fine-tuning with different datasets caused noticcable changes in the circuit components, and this observation held true for both small models (GPT-2 Small) and large models (GPT-2 Large and Tlama-2).
3.4
we analyzed the overlap between

---

Published in Transactions on Machine Learning Research (12/2025)
Lang
Now
Figure 5: Plot Showing Overlap of top edges by EAP scores for Untuned vs Finetuned GPT-2 Small, GPT-2 Large, and Llama-2 in DSSL (Demographic) configuration. Pretrained Pos, Pretrained show positive negative bias pretrained LLMs. Shakespeare Pos and Shakespeare Neg denote positive or negative biased models finetuned on the Stakespeare dataset. Positive pos and Positive neg the underlying circuit the pretrained model finctuned positive dataset.
From Table 3 we can see that corrupting the top edges responsible for bias reduced bias in the original model in most of the cases, except for GPT-2 Large in DSS1. We used the top 400, 1000 and 3000 edges from GPT-2 Small, GPT-2 Large, and Llama-2, respectively. We also wanted to investigate whether this edge corruption also affects the performance the model in other NLP tasks. Consequently, Table 3 shows the performance on two NLP tasks: CoNLL-2003 Sang & De Meulder (2003), a named entity recognition benchmark, and CoLA Warstadt (2019), linguistic acceptability judgment task for all the models. Table shows decrease in performance in CoLA and CoNLL-2003 tasks across all the models. This reduction shows that bias-related edges have an overlap with different language understanding tasks. This differential impact across tasks hints at hierarchical organization of linguistic knowledge within the network, where certain capabilitics are more doeply integrated into these high-inftuential edges than others. Table $ shows the performance of debiased GPT2 Drechsel & Herbold (2025) and Lllama3.2 Drechsel & Herbold (2025) CoNLL-2000 Sang De Meulder (2003), CoLA Warstadt (2019) and VER-CONL2003. It can observed that the decrase in performance in the above mentioned tasks much more compared to our proposed debiasing approach in Table
GPT-2 Small
GPT-2 Large
Llama-2
Qwen-20.5h
Gemma
Model
4 Bine
CoLAA
NRR
Biss
CoLA
NER
Bins
CoLA
NRR
6 Biss
CaLA
NER
Biss
CoL.A
NER
Proposed
88.8%%
22.55
20.40%
8,8956 7
6.139%
9.16%
2.30%
8.25%
36,0156
2.15%
Prspueed
80.57%
18.135
71,00%
4.7%
3.77%
86.00%
0.0%%
1,25%
26,6356
800%
0SS1
Proposed
21 23.8
0.605
2.70%
287% L
6.40%
0.18%
28.84%
5.70%
6.28%
20.31%
1,265
1.21%
31,015
1.13% 1
0.88% 1
0382
Proposed
19 80.86_.1
2,53%
2.83%
1.15%
1000%
1.25%
25.00
7.88%
3.90%
23.12%
2.0%
1030
20.16%
1.00%
0.0%
Table 3: Bias shows the change Bins between the output produced by pretrained model and the model for which blas responsible edges were corrupted. 4. shows decrease in bias and 1 shows increasee in bias. NRF-CoNT.2022,
We also show performance change for CoL.A and
Dehilood GPT-2
Dehiloed Elama
Diehiveed Quom
Dublived Giemms
LEB-CGNLLZIN
NELICGNLLEARE
CEI-CGNLLLam
NEII-CNNLL.2MS
G8S1
Pretrateed
17.9155
9%
35.0%%
5/609 1
4.995
GSSI
Caistin al
36.47%
20 168
1434%
21,155
9.15% 1
6.4% 1
25.0%
RAUS
4.35
59.89%
4.88%
1.58%
Table 4: Performance of Gender debiased GPT2, Llama, Qwen2-0.5b and Gemma2-26 model. Only the pretrained experiment. Llama is done using Tlama3.2 model. However rest of the Tlama experiments are done on Llama2 only. The first row shows results for models trained a debiased model.
6 Conclusion
In this work, we investigated how bias is structurally embedded within the architcctures of GPT-2 Small, GPT-2 Large, LLaMA-2, Qwen2-0.5b, Gemma 2-2b using tools from mechanistic interpretability. Our analysis revealed that, across model scales and architectures, circuits responsible for different categories of bias buch
10

---

Published in Transactions on Machine Learning Research (12/2025)
demographic and gender bias-and largely disjoint, indicating that the underlying representations of bias are modular and specialized. Through targeted interventions specific edges identified via our cansal metries, we demonstrated that it is possible to attenuate bias in model outputs without retraining. However, we also observed that these structural manipulations can negatively impact the model's performance on unrelated NLP tasks, such as named entily recognition and natural language inference. Our findings underscore a fundamental trade-off between debiasing and general task performance, and point to the nced for more selective interventions that can isolate bias-related functionality while preserving broader model competence.
Limitations One of the limitations of this project is that focus specifically on demographic and gender bias in this work. It can be observed from our findings that for different types of bias, the underlying circuit responsible for it will be generally different. Consequently, for other biases, the circuits obtained from this work may not be applied.
References
Alberto Acerbi and Joseph M. Stubbersfield. Large language models show human-like content biases in transmission chain experiments. Proceedings of the National Academy of Sciences, 120(44), 2023.
Abeba Birhane, Atoosa Kasirzadeh, David Leslie, and Sandra Wachter. Science in the age of large language models. Nature Reviews Physics, April 2023.
Tolga Bolukbasi, Kai-Wei Chaug, James Zou, Venkatesh Saligramna, and Adam Kalai, Quantifying and reducing stereolypes in word embeddings. arXiv preprint arXie:1606.00 06121, 2016a.
Tolga Bolukbasi, Kai- Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man to computer programmer as woman is to homemaker? debiasing word embeddinggs Advances in neural information processing systems, 20, 2016b.
Abhijith Chintam, Rahel Beloch, Willem Zuidema, Michael Hanna, and Oskar Van Der Wal. Identifying and adapting transformer-components responsible for gender bias in english language model. arXiv preprint arXiv:2310 1261 2023.
Arthur Commy, Augustine Mavor Pewor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. Advences in Neural Information Processing Systems, 36:16318 16352, 2023.
Jonathan Drecusel and Steffen Herbold. GRADIEND: Monosemantic feature learning within neural networks applied to gender debiasing of transformer models, 2025. URL https://arxiv. /202. 22022 01406
Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Tong Yu, Hanich Deilamsalehy, Ruiyi Zhang, Sungchul Kim, and Franck Dernoncourt. Self-debiasing large language models: Zero-shot recognition and reduction of stereolypes, 2024.
Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. Advances in Newral Information Processing Systems, 34:9574-9586, 2021.
Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 6(3)=e30, 2021.
Nicholas Coldowsky-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching. arXiv preprint or orXiv:2304.0000 2023.
Pengrui Han, Rafal Kocielník, Adhithya Saravanan, Roy Jiang, Or Sharir, and Anima Anandkumar. ChatGPT based data augmentation for impraved parameter-efficient debiasing of LLMss. In Proceedings of the Fourth Workshop on Language Technology for Equality, Diversity, Inclusion, pp. 73 105, March 2021.
HF Canonical Model Maintainers, lbsttlerthhaseuuucccchhheeeteets https://nuggingFace.oom
(revision bfdd146), 2022. URL
base-t---bbe--eccceeeeee -inaae--mm-c------eee--------tte 2-gglissh
Mahaumeed Kamruzzaman, Md. Shovon, and Gene Kim. Investigating subtler biases in LLMs: Ageism, beauty, institutional, and nationality bias in generative models. In Lun-Wei Ku, Audre Martins, and Vivek Srikumar (eds.), Findings of the Association for Compulational Linguistics ACL 2024, pp. 8940 8965, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Tinguistics.
11

---

Published in Transactions on Machine Learning Research (12/2025)
Masahiro Kaneko, Danushka Bollegala, and Naoaki Okazaki. The impact of debiasing on the performance of language models in downstream lasks is underestinated, In Jong C. Park, Yuki Arase, Baolian Hu, Wei Lu, Derry Wijaya, Ayu Purwarianti, and Adila Alfa Krisnadhi (eds.), Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Länguisties (Volume 2: Short Papers), pp. 29 36, Nusa Dua, Bali, November 2023. Association for Computational Linguisties.
Shahar Katz, Yonatan Belinkov, Mor Geva, and Lior Wolf. Backward lens: Projecting language model gradients into the vocabulary space. arXiv preprint arXiv:2402. 00t:2408.1266.. 12865, 2024.
Junsol Kim, James Evans, and Aaron Schein. Linear representations of political perspective emerge in large language models. arXiv preprint Xii:22503. 02000, 2025.
Hadas Kotek, Rikker Dockum, and David Sun. Gender bias and stereotypes in large language models. In Proceedings of The ACM Collective Intelligence Conference, pp. 12-24, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701139.
Samuel Marks, Can Rager, Eric J Michand, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. In The Thärteenth International Conference on Learning Representations, 2025. URL https://openrreviez
Kevin Meng, David Ban, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Adnances in Neural Information Processing Systems, 35:17359- 2022.
Pietro Morerio, Ruggero Ragonesi, and Vittorio Murino. Model debiasing by learnable data augmentation, 2024. URL https://arzie http///axvvvvo.aaaaaa aa////baa////// 049555
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.08 2023.
Pranav Narayanan Venkit, Sanjana Gautam, Ruchi Panchanadikar, Ting-Hao Huang, and Shomir Wilson. Nationality bias in text generation. In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 116-122, May 2023.
Roberto Navigli, Simone Conia, and Björn Ross. Biases in large language models: Origins, inventory, and discussion.
J. Data and Information Quality, 15(2), June 2023. ISSN 1935-1955.
Chris Olah. Mechanistic interpretability, variables, and the importance of interpretable bases. transformer circuits thread (june 27), 2022.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom An introduction to circuits. Distill, 5(3):c00024-00 001, 2020.
Haoyi Qiu, Zi-Yi Dou, Tianlu Wang, Asli Celikyilmaz, and Nanyun Peng. Gender biases in automatic evaluation metrics for image captioning. In The 2023 Conference Empirical Methods in Natural Language Processing, 2023.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
Erik F' Sang and Fien De Meulder. Introduction to the con1l-2003 shared task: langungge-ndeepedeent named entity recognition. arXiv preprint cs/0906050, 2003.
Shweta Soundararajan and Sarah Jane Delany, Investigating gender bias in large language models through text generation. In Mourad Abbas and Abed Alhakim Freihat (eds.), Proceedings of the 7th International Conference on Natural Language and Speech Processing (ICNLSP 2024). pp. 410 424, Trento, October 2024. Association for Computational Linguisties. URL https:://cclanthooogg.... 2ttp:::://ccaambooooooooo///2222 http:/ooor.....2.........e......
Aaquib Syed, Can Rager, and Arthur Commy. Attribution patching outperforms automated circuit discovery. In Youatau Belinkov, Najoung Kim, Jaap Juneleet, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen (eds.), Proceedings of the 7th BlackbouNLP Workshop: Analyzing and Interpreting Newral Networks for NLP, pp. 407-416, November 2024.
12

---

Published in Transactions on Machine Learning Research (12/2025)
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhusale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cyuthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Tsabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibant Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adiua Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliaan Zarov, Yuchen Zhang, Augela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodrignez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL htttps://axxiv wwwpeiiivooggbs///eo wrg..ba///////////o/b//b///////a wttpab/////.09222// 092888
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 33:12388 12101, 2020.
Kevin Wang, Alexandre Variengien, Arthur Commy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprind orxi:2221.000500 2022.
A Warstadt. Neural network acceptability judgments. arXiv preprint orxi::1005.124 2019.
Zhengxuan Wu, Alticus Geiger, Thomas Icard, Christopher Polts, and Noah Goodman. Interpretability at scale: Identifying causal mechanisms in alpaca. In seerth- Conference on Neural Information Processing Systems, 2023. URL httts::/
mett/forru?id-nnfCCinme
Xin Xu, Wei Xu, Ningyu Zhang, and Julian McAuley, Biasedit: Debiasing stereotyped language models via model editing. 2025. URL attps://arxiv.
Fred Zhang and Neel Nanda. Towards best practices activation patching in language models: Metrics and methods, 2024. URL http::///arriv..rrr/a////////// 16042
Ran Zmigrod, Sabrina J. Mielke, Hanna Wallach, and Ryan Cotterell. Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology. In Anna Korhonen, David Traum, and Lluís Marquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 1651-1661, July 2019.
A Appendix
A.1 Description for Biased Dataset
We did an experiment varyting the number of tokens used to computed bias for GPT-2 and Llama-2 and found that beyond 10 tokens the amount of bias present in the model doesn't change that much. Hence we opted for only using 10 tokens to compute bias in LLMs. Table 5 shows the bias computed for each model on different datasets. It also shows the number of instances present in each variation of the dataset along with average length of the input sentences. The temperature is set to 1 for the generation from all models.
A.2 LLMs Explored
We used three different models for our experiments. Each one of them is described as follows.
GPT-2 Small This is a 85M parameter model with 12 layers and 12 attention heads per layer. The model has a dimension of 768 and vocab size of 50257. It uses GELU as its activation function Radford et al. (2019) In its computational graph we have 158 nodes and 32491 edges.
GPT-2 Large This is larger version of GPT having 708M parameters, with 36 layers and 20 attention heads per layer. This one has a dimension of 1280 and vocab size of 50257. Similar to the smaller version it uses GELU as its activation function Radford et al. (2019). In its computational graph we have 758 nodes
and 810703 edges.
Llama-2 This is a 6.5B parameter model with 32 layers and 32 attention heads per layer. The model has a dimension of 4096 and vocab size of 32000. Unlike GPT-2 versions employs SiLU as its activation function Touvron et al. 2023) In its computational graph we have 1058 nodes and 1592881 edges.
13

---

Published in Transactions on Machine Learning Research (12/2025)
Model
Seruct Type
Bins Category
#Samplex
Length
Bias Metric
DSS1
Pusitive
92
7.66
0.639
DSS1
Negative
132
0.753
D5822
Pusitive
141
9.48
0.657
Nepstive
0.633
GS51
Male
293
11.09
0.816
Fomale
G582
Male
291
10.89
0.837
GSS2
29
0.787
DSS1
Positivw
139
7.60
0.786
DS81
Negative
85
0.742
DS82
Positive
60
9.65
0.652
GPT-2 Large
DS82
Negative
0.702
GSS1
Male
290
11.06
0.889
GSS1
Female
30
11.8
0.821
G582
Male
274
10.86
0.847
GS82
Female
46
10.91
0.786
DS81
DS81
Negative
8
6.125
0.718
D582
Lbena-2
DSS2
Negative
8
9.0
0.665
GSS1
GSS1
Female
22
13.36
0.797
GS82
291
0.864
GS82
Fomale
29
13.56
0.790
DSS1
Positive
0.718
DSS1
Negative
6.125
0.632
DS82
Positive
10.0
0,798
Qwvn-2.05b
DSS2
Negative
0.685
GSS1
Male
13.33
0.739
GS81
Fumale
GS522
Male
13.31
0,768
GS82
Funale
0.680
DSS1
Positive
7.0
0.845
DSS1
Negative
DSS2
Positive
10.0
0.851
Gemma
D3822
Nogalive
0.693
GSS1
Male
13.33
0.782
GS81
Fomale
0.695
G582
Male
13.31
0.861
GS82
Fomale
13.55
0.731
Table 5: Dataset Statistics for Different types of Bias Across Different Models. #Sumples- Number of Samples, Length- Avg Length of the sentence. Bias metric Normalized probability of Positive words per example of Postive Bias Category (and similarly the Normalized probability of Negative words per example for Negative Bias Category). Similar stratogy for Gender Bias.
GPT2 Small
0.6
07
0.
0.5
D552
D659
0552
C55
Figure 6: Bias Score Variation With Respect to Top-K values across GPT2 and Llama2
A.3 Baseline Scoring
We have the option calculate the bascline scores for both positivo-bias and negative-bias datascts via two methods i... evalnate-baseiiee scoring and vraluateggrahh scoring. The evalnate-baseline function calculates the difference in probabilities between positive and negative next-token predictions and averages it over the
14

---

Published in Transactions on Machine Learning Research (12/2025)
Node
Deseription
GPT-2 Smalt
GPT-2 Large
Llama-2
Input
Input Node
NA
NA
NA
logits
Logit Node
NA
NA
NA
mill
Ith MLP layer
iE (0-11)
s (0 35)
Ye - 31)
a(i).h(33
jth attention head in
K (0 11);
1 (0 35),
E (0 31),
ith attention layer
j6 (0
J (0- 19)
J (0 31)
x <x>>>>>
<x>>
job [x] attention head
i (0 11);
1 (0 100-00 35),
e (0- 31),
n norry 1kloy, (vlahue
ith attention layer
je (0 11)
JE (0 19)
J (0 31)
Table 6: Nomenclature for Nodes in the computational graph of GPT-2 and Llama-2 (Edges described in Manual Edge Ablation Section Appendix)
whole dataset. While as evaluate-graph function we have the option of passing the argument of Graph (A Graph represents model's computational graph. Once instantiated, it contains various Nodes, representing mostly attention heads and MLPs, and Edges representing connections between nodes. Each Node and Edge is either in the graph (circuit) not; by default, all Node and Edge objects are included within the graph.), and in the process of calculating the baseline score using evaluate-graph function we need to pass the unaltered graph where no edge or node is ablated yet. Since we are going to use the evaluate-graph function when we ablate some edges, hence it is best to use evaluale-graph function for getting the baseline score which we are going to use to check the importance edges.
A.4 Finding Bias Circuits
We sort all the edges in the graph according to their scores and print the edges the descending order with their respective scores for cach positive-bias and nogative-bias datasot. These scores are the respective edge scores and reveal the importance of the edges in the graph for propagating the respective bias. The more the score, the more important is the edge. The goal here to ablate the Top N edges in the graph (N ranging from 1 to 10 in our case) and observe the variation in the evaluate-graph scores and compare it to the graph-baseline score. EAP Syed et al. (2024) eventually outputs sorted list of edges where each edge is represented by the corresponding connecting nodes. In Table D we the description of different types of nodes referred in EAP. Thable 7 shows the top 3 edges obtained for different types of bias across different models (i.e. GPT2- Small, GPTg-large and Llama2).
A.5 Selecting top-K edges
It was computationally infeasible to consider all the edges of a model to do all the analysis. Our primary object was to find the set of edges which were mostly responsible for bias in the model. Hence, we selected the top-k edges, where k represents the least number of odges retained in the model such that the rosulting bias remains within 20% of the original model's bias value. In Figure 7, we varied the total number of edges from 600 to 3000 across different models. The red line in Figure 7 shows the 20% of the original bias value of the model.
A.6 Creating Corrupted Edges for Downstream Tasks
For CoLA dataset we filter the sentence containing atleast one noun, duplicate the sentence and swap every noun token with "XYZ", preserving positions. For CoLA dataset we swap any two randomly chosen words in the sentence.
A.7 Finetuning Experiment Setup
We conducted our experiments using the Hooked- Tranof-r-raffomer framework from the TransformerLeess repository Our base architectures were GPT-2 and Llama-2. Both models were evaluated in their pretrained form and subsequently fine-tuned on two distinct datasets.
15

---

Published in Transactions on Machine Learning Research (12/2025)
45
Willen
Readun Pay.
you Bu
Ovint.ic
tve fual
be fax Wen
20 Drom
61
500gg
or Audl
p0ltt teck
1900ey the
Figure 7: Variation of Top-K edges
16

---

Published in Transactions on Machine Learning Research (12/2025)
Top
Top 2
Top
Model
Bias Type
Edge
Score
Edge
Score
Edge
Score
GPT-2 Small
Demographic Preittve(DSS81)
m11-> Plgits
0.3307
mD- mD-2-m2
0.1900
m0->-m1
0.1858
GPT-2 Small
Demographie (egative(DSSI)
m11-> m1l->logits
0.2655
m0->mm
0.1681
m0->-m
0.1557
GPT-2 Small
Denographic Positive(DSSS2
m 1- logits
0.1580
m0->>m2
0.1205
m9->>logits
0.0921
GPT-2 Small
Denographic (eSS)
m >lgils
0.0720
2-m2
0.0658
m0-2mm
0.0527
GPT-2 Large
Denographic Positive(DDSS)
m35-2-logit logits
0.3188
m33-2-m35
0.1732
m33->-logite
0.1526
GPT-2 Large
Demographic NegativeeDDSS D
logits
0.3219
m33-2-m3
0.1245
a32.h22-2lgiit lngits
0.1123
GPT-2 Large
Demographic DoS8)
m35->llgiis
0.2158
m.92~>logits
0.1226
m33->>m35
0.1070
GPT-2 Large
Demographie (egSt
m35->-logiss
0.1151
mo22>>logits
0.0598
m0->>m
0.0157
GPT-2 Small
Geuder Male(G
impul-2u0
0.0663
juput-2a0.13:0
0.0561
supul- Jupul->=001..(pp
0.0483
GPT-2 Small
Gender Remale(GSS1)
input-2mm
0.0675
input->a0.h55
0.0639
m11->loggis
0.0471
GPT-2 Small
Gender Male(GSS2)
imput-Sm0
0.0615
input->aa.h55pe
0.0602
input->=0.hh5(pt
0.0502
GPT-2 Small
Gender FGSSle
Input->mo
0.0619
loput->a0.
0.0575
mil->-lggis
0.05:11
GPT-2 Large
Geuder MSSe
m35 m13--lloiiss
0.0281
a33.b11 148.h11->logee logils
0.0274
m=33 m633->logits
0.0261
GPT-2 Large
Geuder 1emmle(GSS1]
m33-> =logils
0.0206
a33.hi1 1oolgiissl
0.0203
m333 >logits
0.0192
GPT-2 Large
Gender Male(GSS2)
logits
0.0306
413.h11-2-loghh
0.0265
a32 119.hh2->logge logits
0.0249
GPT-2 Large
Gender (SS2)
m35->-logits
0.0241
a:22.h2--logits
0.0206
a33.h11-2-logit
0.0204
Llama-2
Gender (GSS1)
m31->>logits
0.2467
m.30-
0.0500
m030>>m31
0.0197
Llama-2
Gender FGSS)e(GSS1)
m31->>logiss
0.3141
mlgits
0.1282
m33->>m31
0.0852
Llama-2
Gender MaSe(
m31->>logits
0.2618
m.30- logits
0.0907
m030~>m331
0.05-12
Llama 2
Gender (GS5)
m31.> >logits
0.3308
m330 >logits
0.1461
m30 2-131
0.1095
Table Top edges for Different Models and Different Bias
EAP Va SAE Approach
Po__DSS
DDs__SS2
DSS1
_eg_ DSS2
Male_GSS1
Male GSS1
Male_ GSS2
Female__ GSS1
Fermaee_ GSS2
GPT2 Small
75%
80%
81%
78%
77%
73%
82%
71%
GPT2 Large
73%
72%
71%
76%
85%
75%
72%
81%
Llama2
84%
72%
71%
79%
87%
73%
72%
83%
Table 8: Overlap percentage Between Important Edges Obtained from EAP and SAE.
The first fine-tuning variant used Posétive Dataset, where all countries were paired with positive sentiment tokens. For the gender bias case, male-associated tokens were emphasized. This setup intentionally biased the model toward positive sentiment with the goal of testing whether such polarity shifts could serve as a debiasing mechanism while preserving the grammatical scaffolding of the model.
The second variant used Shakespeare Dalasel, consisting of Early Modern English text. This fine-tune was designed not only to impose stylistic features but also to explore whether forcing the model into a highly structured linguistic domain could mitigate bias. By encouraging the models to adopt Shakespearean vocabulary and syntax, we aimed to examine whether stylistic adaptation reconfigures structural circuits in ways that contribute to debiasing or instead introduces new pathways specific to style.
Training DetailsFineetunil was performed with the AdamW optimizer using a learning rate 10-4 and a batch size of 129. Each model was fine-tuned for 20 epochs on the respective datasets, with early stopping based validation loss. Evaluation was conducted on held-out validation splits, and circuit changes were analyzed using attention weight inspection and edge attribution methods within TransformerLens.
A.8 SAE Approach Results
SAE Marks et al. (2025) is another recent approach to find important components from model similar to EAP Syed et al. (2024). ). In Figure S we have done similar experiments like Figure 2 using SAE. Figure shows that there is not much overlap between demographic and gender bias in general. Table R shows
the overlap between important edges obtained from EAP approach and SAE approach for all the different. variations. It can be seen that the overlap is always more than 70%.
17

---

Published in Transactions on Machine Learning Research (12/2025)
rog,tal
my
140 ge1l
moke 100
with
male
made
krwil.p12
your
matcard
sesitarri
makcand
Figure& 8: Overlap Results from SAE Approach
A.9 Circuit Diagram
Figure shows the circuit diagram for different types of bias (i.e. demographic and gender) obtained from EAP approach Syed al. (2024). A circuit is subgraph of neural network
input
input
a4. h8
a0.h4
m8
m9
m0
logits
logits
(a)
(b)
(c)
(d)
(h)
(k)
Figure 9: Circuit Diagram: (a) GPT-2-SMaal--DSSI-Posii (b) GPT-22-SaaI-DSS2 (c) GPT-2-
Large-DS Large-DSS1-Posee DSS1-Posiiive, (d) 2PT-2-Large Doee-DDSSPPPPiiiin (e)
(f) DSSa--2s-Ss-o
(g) PiSS--Ssall- (h) GPP1-2-Smal--SSS2PPPioviv (i) GPSS (j) GPT-2-
(k) Llama-2-SS1--Pssttivv (I) 2
18


Reviews:
[{"review_id": "9Vpz90LnD1", "reviewer": "Reviewer_DiFn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors discuss the important problem of understanding bias in LLMs, approaching it from the perspective of identifying the specific model edges responsible for such bias. To this end, they define metrics to quantify bias and study how these metrics change as different edges in the model are ablated.\n\n\n## Questions \n1. The authors refer to relevant literature, but one of the most notable papers on dissecting bias in neural networks is the work by Bolukbasi et al. The paper is mentioned in the context of a particular dataset and not the methods (which in many ways share the essence of Mech Interpretability work of using linear probes/SAEs)\n2. Why do the authors choose GPT-2 Llama? The model family and size of course don't matter to the question but was intrigued by the choice since they are quite old and more powerful models of similar or smaller sizes are available. \t\n3. Where do the contexts come from for the Demographic datasets?\n4. I think the authors could clarify their exposition in several places (especailly their results), some of which I mention below\"\n- How were the neutral nationalities—such as “Emirati” and “January to return broadcaster”—identified? Please elaborate on this.\n- How are the edges determined? How many forward passes were performed? It seems that one would need as many forward passes as there are total edges in the model. How is this made computationally efficient? Conversely, please elaborate on how attribute patching is implemented. A clear description would greatly enhance the reader’s understanding of both the methodology and its \n- There are incorrect uses of parentheses that obscure meaning and grammar. For example, on page 6, the statement that “the sum of the probabilities of next-order predictions is greater than or equal to the sum of the probabilities of next-order predictions” reads as a tautology. Perhaps the parentheses should be removed? On a related note, why not generate k sentences and classify them as positive or negative, rather than taking the original sentence? Wouldn't that make a more biased (which I guess is the desire here) dataset?\n\n\n## Concerns\nI think the above can be easily fixed. I am more concerned about the broader methodology:\n\nThe first major point relates to the stability of the findings. It appears that the identified bias edges are unstable: as the authors note, fine-tuning leads to a different set of edges than those originally found. Moreover, the edges vary across different tasks—for example, demographic bias edges differ from those associated with gender bias, and the same holds for grammatical reformulations. \n\nGiven the above, I find the claim on page 7—that “these results suggest that the circuits for bias demonstrate similar generalizability patterns across both small and large models”—to be quite misleading.\n\nThe authors also find that attempts to mitigate bias lead to a decrease in performance on other natural language tasks. Further suggesting the lack of a clear set of edges and potential confounders. On a related note, what does \\delta in Table 3 represent, and how is it defined?\n\nGiven these findings, it almost feels like p-hacking—though I am sure that this is certainly not the authors’ intention. In a sufficiently large network, one can almost always identify some subset of edges that appear more or less important for a given task!", "claims_and_evidence_yn": "No", "claims_explanation": "See above.", "audience_yn": "Yes", "audience_explanation": "I think the question phrased by the authors is very interesting and of interest to TMLR.", "requested_changes": "See above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hBf37jtWhL", "reviewer": "Reviewer_tbkQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- uses Edge Attribution Patchingto localize edges in LLM, gpt2 small/large and llama2, that carry demographic and gender bias.\n- proposes two novel (general-context) semantic bias metrics.\n- studies the effect of variations of corruption techniques.\n- demonstrates localization of bias to a relatively small set of layers.\n- analyzes the stability of important edges with respect to w.r.t. 1. Grammatical Structures, 2. Different Types of Bias, 3. Finetuning\n- leverages identified important edges to debias LLMs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The experiments are reasonably detailed given the computational resources. The methodology is intuitive and results generally support the central claims.", "audience_yn": "Yes", "audience_explanation": "Researchers in mechanistic interpretability, AI safety, fairness/privacy, and the broader ML community will find this problem interesting, as this is a trending topic nowadays. The paper provides concrete evidence with the lens of MI that (i) bias concentrates in relatively few layers, (ii) they also affect LM's general capabilities, and show (iii) various stability results. These are useful starting points for both empirical MI and fairness work.", "requested_changes": "- in the second last paragraph on page 7, \"... edges in $DSS1_{pos} (GSS1_{pos})$ and $DSS1_{neg} $ $GSS1_{neg}$, in Figure 2...\" should be \"$DSS1_{pos} (GSS1_{pos})$ and $DSS1_{neg} (GSS1_{neg})$\" right?\n- in the first paragraph of section 3, \"An edge refers generally represents...\"\n- in table 4, the column name is Debiased Llama2 but the caption is llama3.2\n- the writings, in particular the clarity, should be substantially improved", "broader_impact_concerns": "N/A", "additional_comments": "Metrics:\n- I assume the temperature is 1 throughout the experiment. Is that correct? It would be great to mention it in the appendix A.\n- Both metrics proposed rely on externally defined positive/male and negative/female sets. I’m especially concerned about the sentiment proxy using distilled-bert. As demographic-bias conclusions depends on the truthfulness/validity of the sentiment proxy, please justify this choice.\n- In appendix A, you claimed that \"beyond 10 tokens the amount of bias present in the model doesn’t change that much\". I wonder if you have can provide a simple plot showing the effect of k when $k \\in [1,10]$ and  $k \\in \\{20, 50, 100\\}$ to illustrate the effect of $k$.\n- In table 2, you showed that $(L_2,C_2)$ leads to the smallest changes in metric, so you choose them for the future experiment, since \"smaller change implies that set of important edges identified by EAP performs almost similarly as the whole model where all the edges have been considered\". I wonder if the two metrics inherently have different sensitivity. Intuitively $L_1$ is more sensitive than $L_2$, as $L_1$ is two-sided, so choosing the smallest raw changes in metric might lead to choosing the least sensitive metric instead. Hence, comparing the normalized changes would make more sense to me, where the normalization takes the sensitivities into account. I hope I make my point clear. In fact, I also observed that the L1's changes are significantly larger that that of L2, which supports my concerns.\n\nFinetuning experiment:\n- it would be good to provide the training recipes in the appendix.\n- why these two datasets? Could you provide a brief justification? I would imagine a suite like glue or similar might be a better choice?\n\nDebiasing:\nI do not quite understand the arguments on page 9 (and figure 3/4.) In particular, is the number the relative percentage changes compared to the pretrained model? Is it the case that figure 3 shows your debiasing approach and that figure 4 shows Drechsel and Herbold (2025) approach? Is the model you presented in figure 4 llama2 or llama3.2 (I assumed it's a typo)? Is the eval recipes the same throughout all the evaluations, e.g. sampling strategy + hyperparams, prompts? Why is the number in $\\delta$ bias differ so significantly, gpt2-large DSS2 v.s. the others? Probably it would be good to also provide the actual numbers if those percentages are relative.\n\nOthers:\nIt is very interesting that for demographic bias under grammatical variation, the sets of important edges exhibit minimal overlap, while there exists a substantial degree of overlap for gender bias. It would be great if you could expand discussion and hypothesize mechanisms a bit more. Also, it feels worthy to promote a compact version of Appendix F (SAEs) to the main text to triangulate your causal story and compare against an alternative approach.", "extra_scores": {}, "extra_text": {}}, {"review_id": "xz0eTJ4iLf", "reviewer": "Reviewer_1xiq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors adopt a mechanistic interpretability(MI) perspective and examine demographic and gender bias in GPT-2 Small, GPT-2 Large, and Llama-2. This paper is motivated by the limited generalizability of prior debiasing approaches that primarily rely on fine-tuning or data augmentation; The authors share analysis on these previous methods as well. Using Edge Attribution Patching (EAP), they investigate whether biases are localized within distinct substructures of these models.  \n\n\n**Key findings:** \n1. Demographic and gender bias are largely localized to certain edges and layers across models. \n2. The location of bias-related edges in a network tends to shift  under perturbations (e.g., input variation, fine-tuning), rather than remaining consistent across conditions.\n3.Debiasing by corrupting these edges reduces bias but also leads to a trade-off, lowering performance on downstream tasks. <br>\n\n\n\n**Strengths:** \n1. It addresses an important problem of debiasing LLMs, with a particular focus on whether bias can be localized within specific components of the model.\n2. The paper leverages Edge Attribution Patching (EAP), an existing method for estimating the causal importance of edges with minimal computation (two forward and one backward pass).\n3. The experiments are extensive. Experiments examine bias under diverse corrupted sample settings, evaluate stability across different fine-tuning settings, and compare results across models of different sizes and architectures. \n\n\n\n**Weaknesses:** \n1. Despite criticizing prior work for narrow scope, this paper also focuses only on two bias types and two architectures, leaving its generalizability uncertain.\n2. The novelty of this paper appears limited. First, the methodology used—Edge Attribution Patching (EAP) and the L1/L2 bias metrics—relies on previously introduced techniques. Second, the central conclusion that bias is localized to specific components of LLMs is not entirely new, as similar claims have been made in prior work [1,2].\n3. Some results are difficult to interpret, e.g., Table 2 shows only marginal differences without defining “minimal change,” and Figure 3’s localization evidence is less convincing for Llama-2 than for GPT-2 Small.\nThe study is limited to GPT-2 and Llama-2, raising concerns about the applicability of findings to more recent models. \n\n\n[1] Jesse Vig et al. Investigating Gender Bias in Language Models Using Causal Mediation Analysis. In Advances in Neural Information Processing Systems (NeurIPS), 2020. \n\n[2] Yi Yang et al. Bias A-head? Analyzing Bias in Transformer-Based Language Model Attention Heads. arXiv preprint arXiv:2311.10395, 2024.", "claims_and_evidence_yn": "No", "claims_explanation": "The experiments are thoughtfully designed, but the evidence is not always convincing. The use of outdated models (GPT-2 and Llama-2) makes it unclear whether the findings would generalize to more recent LLMs. Some results (e.g., Table 2 and Figure 3) are ambiguous and not sufficiently justified, making the localization claim less robust. Additionally, the scope is limited to only two bias types and two architectures, which narrows the findings of this work despite the paper’s stated motivation of generalizability.  The description of the approach taken is also not specified enough to clearly understand pros and cons of the proposed mechanistic approach.", "audience_yn": "Yes", "audience_explanation": "The paper addresses a timely and important issue: understanding how bias is represented within LLMs through mechanistic interpretability. Even if limited in scope, the idea of bias localization at the edge/layer level and the use of EAP for analysis would be of interest to researchers working on LLM bias, interpretability, and safety.", "requested_changes": "## 1. Details required for understanding overall experiments\n\n\n**1-1. Corrupted samples**: Section 3.2 should provide more detail on how corrupted samples were constructed. While the paper cites prior work as the source of neutral terms, it remains unclear how many neutral replacements were sampled and what selection criteria were used. Clarifying this would improve transparency.\n\n\n**1-2. Bias metrics**: Section 3.1 states that higher values of L1 indicate stronger bias. However, since L1 can be either positive or negative depending on the favored group, this phrasing is misleading. It would be more accurate to state that higher absolute values of L1 correspond to stronger bias, with the sign reflecting the direction of bias.\n\n\n**1-3. Clarification on “Edge” Definition**: The definition of an “edge” is somewhat abstract. Please provide more concrete examples of what an edge represents in the computational graph (e.g., outputs of MLP blocks or attention heads). \n\n\n**1-4. Definition of important edges**: In Section 5, Table 2 compares “important edges” with all edges. While the paper defines importance within the EAP framework, it does not specify the threshold criterion for classifying important edges. A more precise description is required to ensure reproducibility.\n\n\n## 2. Missing technical details in methodology section\n\n\n**2-1. EAP explanation**: EAP is a central methodology in this work and is claimed to be more efficient than naive ablation, requiring only two forward and one backward pass to compute importance scores for all edges. However, based on the formula provided, it is difficult to understand why such efficiency is achievable. While a full derivation may not be necessary here, a brief high-level explanation or more explicit reference to prior work would improve clarity for readers unfamiliar with EAP.\n\n\n**2-2. Details on Gradient Computation for EAP with External Metrics**: The paper does not clearly explain how gradients of a loss defined by an external sentiment model (e.g., DistilBERT) are backpropagated to the LLM’s internal edges. Please clarify how differentiability is maintained across discrete token generation and whether any special techniques are used.\n\n\n## 3. Miscellaneous\n\n\n**3-1. Comparison with other debiasing methods**: The experiments only include one debiasing baseline. It would strengthen the paper to either compare with other established debiasing techniques or clarify why such comparisons were omitted.\n\n\n**3-2. References**: On page 3, Syed et al. (2023) and Syed et al. (2024) are cited separately, but they appear to be essentially the same work on attribution patching (same authors and overlapping content). Unless there is a substantive difference in how these two papers are intended to be used as references, they should be combined into a single citation.", "broader_impact_concerns": "No major broader impact concerns to note.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)