Multi-Feature Quantized Self-Attention for Fair Large Language Models
Authors: Jaeil Park
Sung-Bae Cho
Venue: ICLR 2026 Poster (2026)
Keywords: Large language models
multi-attribute social bias
quantized adversarial autoencoder
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Large language models (LLMs) often encode social biases tied to sensitive features such as race and gender, undermining fairness in downstream tasks even after instruction tuning. Conventional debiasing methods require expensive fine-tuning, are tied to specific architectures, or operate only at the input or decoding stage while neglecting attention-level representations, which can result in compromised task performance. Moreover, most approaches are tailored to single-attribute settings and do not explicitly address scenarios with multiple, overlapping protected attributes and their intersections. This paper proposes a novel method of multi-feature quantized attention regularization (MQAR) to mitigate multi-feature bias by injecting a structured quantization into frozen self-attention layers. MQAR disentangles attribute-specific activations through vector-quantized regularization and uses a discriminator-guided autoencoding regularizer to adversarially suppress protected-attribute information while preserving task-relevant semantics. Crucially, the proposed method operates without modifying the backbone parameters or accessing pre-training data, ensuring architecture-agnostic applicability and minimizing representation distortion. MQAR is evaluated on five diverse LLMs (BERT, T5, GPT-Neo, Mixtral, and LLaMA 3.2) using three standard bias benchmarks (WinoBias, StereoSet, and CrowS-Pairs). Across these models, MQAR consistently reduces bias for multiple protected attributes and their intersections while maintaining downstream accuracy within at most 0.4 \%, on average, of non-debiased baselines on sentiment analysis, abusive language detection, and text generation tasks. These findings highlight quantized attention regularization as a scalable and effective method for mitigating social bias in modern language models.

Published as a conference paper at ICLR 2026
MULTI-FEATURE QUANTIZED SELF-ATTENTION FOR FAIR LARGE LANGUAGE MODELS
Jaeil Park & Sung-Bae Cho
Department of Computer Science
Yonsei University
1603, sbcho @yonsei. ac.kr
ABSTRACT
Large language models (LLMs) often encode social biases tied sensitive fea- tures such as race and gender, undermining fairness in downstream tasks even after instruction tuning. Conventional debiasing methods require expensive fine-tuning, are tied to specific architectures, operate only at the input or decoding stage while neglecting attention-level representations, which can result in compromised task performance. Moreover, most approaches are tailored to single-attribute set- tings and do not explicitly address scenarios with multiple, overlapping protected attributes and their intersections. This paper proposes novel method of multi- feature quantized attention regularization (MQAR) to mitigate multi-feature bias by injecting structured quantization into frozen self-attention layers. MQAR disentangles attribute-spen specific activations through vector vector-quante quantized regularization and uses discriminator-guiin autoencoding regularizer to adversarially sup- press protected-attrine Crucially, the proposed method operates without modifying the backbone pa- rameters or accessing pre-training data, ensuring architecture-agnosti applica- bility and minimizing representation distortion. MQAR is evaluated on five di- verse LLMs (BERT, T5, GPT-Neo, Mixtral, and LLaMA 3.2) using three standard bias benchmarks (WinoBias, StereoSet, and CrowS-Pairss Across these models, MQAR consistently reduces bias for multiple protected attributes and their inter- sections while maintaining downstream accuracy within at most 0.4 %, on aver- age, of non-debiased baselines sentiment analysis, abusive language detection, and text generation tasks. These findings highlight quantized attention regular- ization scalable and effective method for mitigating social bias modern language models.
information while preserving task-relevant semantics.
INTRODUCTION
Large language models (LLMs) have shown remarkable performance across a wide range of natural language processing (NLP) tasks. By leveraging self-attention mechanisms, L.MMs encode rich contextualized representations that can generalize well to downstream applications such as sentiment analysis, natural language inference, and text generation (Sun al. 2019 Alaparthi Mishra [022]] Lin & Sul 2021 Zhu et al. 2020). However, it has been widely observed that these models also absorb and propagate social biases present in the training data, including those related gender, race, and religion (Schramowski et al., 2022). Such biases not only undermine faimess but also introduce systemic risks when LLMs are deployed real-worle applications.
Recent work has shown that bias in LLMs is not solely inherited from data but is often amplified by the internal self-attention structures of the models (Jiagg al. (022). The attention mechanism entangles sensitive feature information with semantic content, causing the model to produce biased outputs even seemingly neutral contexts (Liu et al.) 2024). For example, detection, sentences containing female-related terms are more likely to be misclassified as offensive, reflecting a learned association between gender and abuse PPak et al., 2018 Park Cho 2025). These representational biases persist even when downstream classifiers trained separately, the bias embedded within the latent representations themselves, and they remain measurable even in instruction-tune and alignment-optimizee I.LMs.
abusive language

---

Published as a conference paper at ICLR 2026
Debiasing LLM
Debiasing Downstream Task
LLM
Wa
Wo
LLM
Dataset
Self-attention
Dataser
Sentence
Wi
Debiased
wi
a
quantization
debias
Wn.
Wii
Figure Overview of the proposed MQAR method. Token sequences are first encoded via the frozen self-attention layers of a pre-trained LLM. Structured quantization is applied to the attention outputs to isolate attribute-ppecific activations, followed by an adversarial autoencoder that regular- izes the latent space to remove biased information without compromising semantic consistency.
To address this, several debiasing methods have been proposed, including dataset reweighting (Mozafari et al. 2020). input perturbation (Manerba & Tonelli 2021). and salrness-awart fine-tuning Yu et al. 2023). More recently, inference-time techniques based on prompting and decoding have also been explored
steer model outputs without modifying model parameters. However, these approaches suffer from two key limitations. First, they often rely on retraining or fine-tuning the backbone of the LLM, which computationally expensive and often architecture-dependent Sec- ond, many methods are designed for a single protected attribute and do not scale well to real-world scenarios involving multiple, overlapping protected attributes and their intersections (Zhao et al,, 2023). Consequently, there remains need for model-agnostic, lightweight debiasing method that directly targets representation-leve bias in multi-attribute settings without modifying the core parameters of the LLM.
To overcome this problem, this paper proposes Multi-feature Quantized Attention Regularization (MQAR), novel method for mitigating social bias in LLMs by intervening in their frozen self- attention layers. As shown Figure [ MQAR introduces a structured quantization process into attention outputs to disentangle attribute-specife activations, followed by adversarial autoen- coder that regularizes the representations to suppress bias while preserving semantic information. In contrast to prior approaches, MQAR operates directly the internal attention representations of a frozen backbone and is trained with multi-feature supervision so that a single MQAR module can jointly debias multiple protected attributes (e.g., gender and race) without access the original training data. Moreover, MQAR leaves the LLM backbone untouched, enabling generalized de- ployment across diverse pre-trained models, including BERT, T5, GPT-Neo. Mixtral, and LLaMA 3.2.
MQAR is evaluated on two widely used bias benchmarks, WInoBias, StereoSet. and CrowS-Pain Across five LLMs, MQAR consistently reduces bias for multiple protected attributes and their in- tersections, while maintaining downstream accuracy within at most 0.4% of non-debiased baselines on tasks such as sentiment analysis, abusive language detection, and text generation. These results position MQAR as scalable, vichitectrre-annottcc and fine-tuning-ffe debiasing solution. Its ability to mitigate multi-attribute bias without retraining makes it especially practical for real-world applications where retraining is infeasible.
RELATED WORKS
2.1 LARGE LANGUAGE MODELS AND THEIR BIAS
Transformer-basee LLMs such as BERT (Devlin et al. 2018). T5 Raffel al. 2020). GPT-3 Brown et al. 2020 Mistral (Jiang et al. 2023), and LLaMA Touvron et al. 2023) have achieved remarkable performance on a wide range of NLP tasks. These models learn rich contextual represen- tations through self-attention over large-scale corpora, which inherently reflect and reproduce social biases present in the data (Schramowski et al. 2022). Such biases have been shown propagate to
2

---

Published as a conference paper at ICLR 2026
downstream applications, leading to discriminatory behavior in practical tasks such abusive lan- guage detection, sentiment analysis, and coreference resolution, and they remain measurable even in instruction-tune tuned or alignment-optimizze variants.
To evaluate these effects, several benchmark datasets have been developed. SEAT May et al. 2019)
measures sentence-level association bias, WinoBias (Zhu et al. 2020) focuses gendered corefer- ence resolution, StereoSet (Nadeem et al. 2020) and SHowS-PPirs (Nangia et al. (020) target stereo- typical associations across domains, BiasBios (De-Arteaga al. using gender-masked bios, BBQ (Parrish et al. 2022) explicitly probes social and intersectional bias in question answering settings, and CoBia (Nikeghbal al. 2025) constructs lightweight adversar- ial conversations in which an LLM first produces biased claim. These datasets collectively provide a comprehensive suite for evaluating representational and behavioral biases L.LMs under both static and interactive settings. This paper primarily employs WinoBias, StereoSet, and CrowS-Pairs to assess how well MQAR mitigates biases across multiple protected attributes.
2019) evaluates occupational bias
2.2 BIAS MITIGATION IN LANGUAGE MODELS
Bias mitigation strategies for language models are typically categorized into embedding- model-level, and inference-time approaches. Embedding-based methods, such as SentDebias (Liang) ctt al. 2020), INLP (Ravfogel et al. 2020), and OSCaR (Dev et al., (2020), aim to remove protected- attribute information via projection (e.g., removing directions aligned with protected attributes) or representation alignment. However, these methods may compromise the expressiveness of learned features by overly constraining the latent space, thus reducing its capacity encode task-relevant information (Shin et al.) 2020; Kaneko & BBlleggla 2019).
Model-level approaches. including ADELE (Lauscher al. 2021 and FaRM (Chowdhury & Chaturvedi 2022). inject fairness objectives into training loss functions or introduce debiasing mod- ules during training. While effective, such methods often require access training data and exten- sive fine-tuning of the backbone, which limits their scalability and adaptability to new domains and proprietary LLMs.
Inference-time approaches aim to mitigate bias during model inference by manipulating prompts or controlling decoding strategies (Fatemi al.) (02T). and more recent methods such as CRISPR, CPAD, and RB (c.g.. Yang et al. [2024] Dai et al., 2024 Kim et al. (2025) further refine outputs by adjusting generation trajectories without changing model parameters. These techniques are attrac- tive because they are model-agnostic and freem,ter-free, but they do not directly modify internal representations, and therefore may leave attention-level biases intact.
Despite recent progress, most prior work targets single protected attribute at a time and pro- vides limited support for scenarios involving multiple, intersecting protected features (e.g., gender and race) (Zhao et al. 2023). Moreover, existing methods are typically tied either to embedding- level projections, full model fine-tuning, or purely output-level inference- time control. In contrast, MQAR directly targets internal attention activations in frozen LLMs by introducing quantized bot- tleneck and an adversarial autoencoder that jointly debias multiple protected features with a single plug-in module, without access pre-training data modification of backbone parameters. To the best of our knowledge, MQAR is among the first attention-level debiasing methods that operate entirely on frozen LLMs while explicitly addressing multiple protected attributes and their intersec- tions.
QUANTIZED REGULARIZATION OF SELF-ATTENTION
3.1 REPRESENTING SELF-ATTENION OF LLMS
To formalize the intervention point for the proposed regularization, this section begins by briefly reviewing the self-attention mechanism in transformer- LLMs. Given an input token sequence W. {Wa. ... w, pre-trained LLM encodes it through stack of multi-head self-attention layers, producing contextualized representations at each layer. Let X;- e RR xd denote the hidden state at layer 1. The attention operation at layer i is defined as:
3

---

Published as a conference paper at ICLR 2026
[CL5]
MatMul
you
you
are
Sofhwax
pood
good
Mask (opt.)
weman
woman
[SEP]
Scale
Training the quantized autoenceder and discrinninator
KLS
(CLS)
MatMuul
you
you
0
K
goori
good
Regularization
man
man
[SEP)
Regularization by the discriminator
(5)
Figure 2: (a) Visualization attention heatmaps in BERT for sender-swapped input pairs, reveal- ing differential focus induced by gender bias. (b) Schematic overview of MQAR's integration into the self-attention layer via pre-attention regularization. (c) Conceptual diagram illustrating how quantized regularization compresses and disentangles protected attribute signals from semantic rep- resentations.
[QKKT Vd
X; softmax
V4,
Q: = 0 K; X. == X (1)
where dę denotes the dimension of the key vectors. While this attention mechanism effectively cap- tures contextual dependencies, it can also entangle task-irrelevant signals such as sensitive feature information. As illustrated in Figure 2(a), gender-swapped sentence pairs elicit markedly different attention maps, indicating that the attention mechanism encodes attribute-specific signals.
To mitigate this, illustrated in Figure c), this paper introduces layer-wist quantized regular- ization (QR) module that transforms the input X;-1 into debiased representation X;-1:
= -R--I(-(XX-))
W X
X; softmax
(3)
dd
Section 3.2 describes formal explanation of the regularization. This design enables MQAR to suppress sensitive feature information within the attention computation itself, improving fairness without degrading contextual expressiveness. Crucially, the method requires no modification to LLM parameters, making it applicable to frozen pre-trained models.
3.2 REGULARIZING SELF-ATTENTIN WITH QUANTIZED ADVERSARIAL AUTOENCODER
The proposed method mitigates representational bias in self-attention by applying structured reg- ularization over latent representations within quantized adversarial autoencoder. Autoencoders compress high-dimensional embeddings into lower-dimensional latent space, and when combined with adversarial learning, allow suppression of sensitive attribute signals while retaining semantic content.
Structure. Formally, given a token representation I(t.)) at layer and position i, an encoder é first maps it to latent vector EQ,4) = e1 (zq..)) A reconstruction decoder dj then produces.//..) di (zus)) and a reconstruction loss encourages *((.)) to remain close to W(I.)), ensuring that 20.0) preserves the original semantic content. To introduce debiasing bottleneck, MQAR applies a structured vector quantizer Q to obtain mutated latent representation r(t.)) =Q(=q.)) (= which
4

---

Published as a conference paper at ICLR 2026
is fed forward to subsequent layers in place of the original representation. A second decoder d2 reconstructs the token representation from this quantized latent =(I,,) d2 (T(...)) , and an additional reconstruction term encourages T(I.)) = (1,2) so that quantization and debiasing do not excessively distort task-relevant semanties.
For debiasing, each token (or sentence) is annotated with a multi-hot protected-attrinee label vector E {0, 1)MM, where M is the number of protected attributes (c.g.. gender, race, religion) and each dimension indicates the presence of a particular attribute. A discriminator D, takes the mutated latent T(..)) input and predicts å (r(,.)))) During training, Da is optimized correctly classify while the multi-hot supervision discourages the model from encoding correlated attribute signals a shared latent subspace and drives T(..)) toward a representation that retains semantic information but sup- presses multiple protected features jointly.
pair (e1. Q) is trained adversarially to obfuscate This
Accordingly, the quantized regularization (QR) module at layer and position i can be viewed the composition (-) = d2 (Q)). which maps an attention output to its debiased, reconstructed counterpart before it passed to the remaining layers.
Training 1: Training quantized autoencoder and discriminator. According Hsu al. (2024), the proposed method uses two types of loss to train quantized autoencoder, consisting of encoder 61 and decoder d1, in differentiable manner. The two losses enable the autoencoder to learn the discrete representation à E ZX of a in the latent space through the data and to guide the latent vector of the input data towards à with some integer K and m'. Here, tized latent vector derived from a codebook of size K, representing discrete prototypes aligned with protected attributes. The encoder e₁ maps inputs continuous latent z, which is then quantized into à using nearest-neighbor search. The straight-through gradient estimator enables the propaga- tion of gradients through the non-differentiable quantization step. The quantization loss Lygentize pulls the discrete values constituting unilaterally towards the encoder's 's pre-quantized continuous output. This adjustment is necessary to optimize because the straight-through gradient estimation decouples à from the computation graph. In contrast, the commitment loss Leommit ensures that the pre-quantized representation, which receives gradients from downstream computations, remains close to the codebook entries. Although this issue is significant in vector quantization, using scalars instead of high-dimensional vectors mitigates the problem. This allows for a substantial reduction in the quantization and commitment losses while maintaining training stability, providing the model with essential flexibility to reorganize the discrete latent space.
Z denotes the quan-
Lquantize [|StopGradient(2) all2
(4)
Lcommit = ||= StopGraddent(())1
(5)
Training of the autoencoder, decoder di, d2, and discriminator occurs simultaneously. The losses for the decoder and discriminator, which are composed of MI.Ps, use BCE (Binary Cross-Entropy) to compare with æ and respectively. Equation (6) represents the overall loss term this stage.
L = Lquantize + Leommit + Ld, + Ldd + LD.
(6)
Training 2: Regularization by discriminator. Subsequently, the discriminator D, is fixed, and {au..... am are randomized a"
protected feature labels of the mutated latent vector (a5:...,"", }. As = and r are regularized, while D., maintains to discriminate the protected feature label of them appropriately, and are learned not to distinguished by D.. By iterating learning the discriminator and randomizing the label, and are learned to disentangle for protected features (Makhzani et al. 2015 Park & Cho. 2023).
The goal is to optimize the latent representation such that it maximally retains information about the input sentence I, while minimizing the mutual information with protected attributes a. This objective aligns with the information bottleneck principle, and is formally expressed as equation (7). where I represents mutual information, X is the original embedded representation, R the mutated latent vector, A is the attribute of protected feature, and /3 is a coefficient to balance the two terms.
5

---

Published as a conference paper at ICLR 2026
max =
-
(7)
To deal with the difficulty in estimating mutual information, this paper uses upper and lower bounds along with Monte Carlo gradient estimates. This paper introduces the following lemmas, and the proofs for them are presented in Appendix
As maximizing I (R; X) represents the association value between the original token X and the latent vector R, we can maximize the lower bound of equation (8)
Lemma 1. For any conditional distribution p.
[ + ((x)
where D2 is the function of decoder 2 d2, q is the encoder function, p is the real distribution, m the projection function, and H is the entropy.
As entropy non-negative, the lemma implies the lower bound equation 9
(R
"a.)) (d...)(((..)) (-m (x)
== Lr
(9)
At the same time, by minimizing (Rt; A) which represents the association value between the latent vector R and the protected feature attribute A, we can also maximize the lower bound of equation 7
Lemma 2. For any conditional distribution m,
m(rl2) = Ci s(r)
TRR; A) S mm (rlz))
(10)
However, it is not ideal because considering only the function is not stable. For the tighter upper bound when assuming optimal dual regularization, T(R; A) can be approximated.
Corollary 3. I'Dxr.((aap & (alr)) S I,
I (R;; A) Eptra) [log (a r) - log (a)] [a)]+[
(1I)
As the adversarial discriminator approaches the global optimum, that is, 0, the upper bound [[R; A) can be written as equation (22
Algorithm 1: Latent Regularization via Adversarial Training
Data: Data X and protected features A Result: Fully trained encoder 61
I (R; A) < Ep(r,a) (a r) - log (a)] C2
(12)
Initialize e1, D1, D2;
for epochs do
The final objective function for bias mitigation can be formulated as equation (7).
for batches do
Sample x a, ä from X, A,
U00 1) respectively;
max C, - B1C -22C2
(13)
0g 0, 1) 36. ac_
aC. (Ir, a);
As the autoencoder is optimized the distribution space, this paper can show that strong duality holds within a constrained optimization approach (Song
0p. pp. 180 (z, a);
0.. + 0g - 00
a(Ci+ (æ, a);
al. 2019).
0m + 0m 11
81C1 +(C1+0a) (x, a);
80
Theorem If 61,62,63 f3 > O, then strong duality holds for the following optimization problem over distributions p.e.m min L s.1.C, < <x1. <22 p.
0p, t 0D, 18
aC2
(x, a):
end
end return ei;
(14)
While L, encourages and to retain maximal information about the input x, the regularization terms C₁ and C2 act as adversarial constraints, discouraging encoding of protected attribute infor- mation. Algorithm shows its training process.

---

Published as a conference paper at ICLR 2026
3.3 USING DEBIASED LLMS FOR DOWNSTREAM TASKS
MQAR functions as a lightweight plug-in module applied at the self-attention level of a pre-trained LLM, transforming token representations before they are processed by the frozen layers. The trans- formation preserves semantic content while suppressing protected attribute signals. Once debiased, the modified embeddings are propagated unchanged through the rest of the model. Since MQAR does not require access to backbone weights or gradient updates, it is well-suited for deployment in black-box models and resource-constrainee environments.
To train the adversarial discriminator Da. MQAR adopts weak supervision from curated lexicons (Dev et al. 2020), selecting only instances that contain clear linguistic cues of protected attributes (e.g,, pronouns, occupational terms). Ambiguous or neutral instances are excluded to prevent su- pervision noise and maintain precise regularization. This selective supervision ensures that bias mitigation is applied only where necessary, avoiding distortion of unbiased content.
Unlike prior debiasing approaches that rely on model fine-tuning, data augmentation, or training- time fairnsss constraints, MQAR operates entirely during inference. It introduces structural modifications and requires no retraining, making it a practical and scalable solution for fairness- sensitive applications.
4 EXPERIMENTAL RESULTS
4.1 EXPERIMENTAL SETUP
For experiments, this paper has conducted all experiments on Ubuntu servers equipped with Intel(R) Xeon(R) Silver 4210R CPUs and four NVIDIA A100 GPUs. For each backbone LLM (BERT, T5, GPT-Neo, Mixtral, and LLaMA 3.2), this paper follows the original implementation and hyperpa- rameter settings reported in the corresponding papers unless otherwise noted. The MQAR autoen- coder configured so that its input and output dimensions match the hidden size of each LLM, and the latent dimension is set to 400. This paper optimizes MQAR with learning rate of 1 x 10-1, apply early stopping with a patience of 7 based on validation loss (checked every 30 epochs), and keep all LLM parameters frozen.
MQAR is trained with multi-hot protected-attribute labels (e.g., gender and race) in the adversarial discriminator to jointly debias multiple attributes within single model. Unless stated otherwise, this paper reports the mean and standard deviation over three random seeds for all metrics, and this paper uses paired t-tests to assess the statistical significance of improvements over baselines. Detailed hyperparameters and additional implementation details are provided in Appendix
4.2 RESULTS ON BIAS IN LLMS
To measure bias in LLMs, this paper uses WinoBias (Zhao et al.) 2018 StereoSet (Nadeem al. 2020), CrowS-Pairs (Nangia et al. 2020). This section compares the raw LLMs against a wide range
of debiasing methods, including representation-level approaches (INLP (Ravfogel
al.) 2020).
SentDebias (Liang et al. 2020)) model-level methods (ADELE (Lauscher et al., 2021]. FaRM al. 2023 recent deep debiasing mod- ules (HDD (Zayed et al. (023)). and inference-time debiasing techniques (CRISPR(Yann) (ang et al.,
(Chowdhury & Chaturvedi 20022). FineDeb FSaeaen(Saravanan
2024. RB(Kim et al.) 2025)), addition to the MQAR. Appendix summarizes the datasets and
metrics in detail.
Figure reepors WinoBias results across five LLMs. Pro (pro-stereotype) and Anti (anti-stereotype) denote the intensities of pro- and anti-stereotypical preferences, respectively. Avg. is their average, and |Difl the absolute difference, which this experiment uses as a bias score, As shown, INL.P, SentDebias, ADELE, FaRM, FineDeb, HDD, and MQAR all reduce stereotypical associations com- pared to the raw LLMs. Among them, MQAR achieves the lowest Avg. and a notably small [Dffll indicating that it better balances stereotype reduction with context preservation.
Table summarizes results on StereoSet and CrowS-Pairs for LLaMA 3.2 with different debias- ing methods, results for other backbones are provided in Appendix C and results with standard deviation are provided in Appendix [ Representation-Leww methods such as INLP and SentDe- bias reduce the stereotype score (SS) and bias CrowS-Pain but often yield suboptimal ICAT or
7

---

Published as a conference paper at ICLR 2026
Pro. Anti
Avg.
[PV/]]
Pro.
Ant.
AVE
101/11
D0RT
20
20
CIT GPP--Me0
INLP
SenTabbix
15
ADELE
FaRM
Limm3 2
FindDab
HDO
Mosirl
Our
20
Figure 3: Experimental results of WinoBias. Pro and Anti show intensity pro-stereotype and anti-stereotype, respectively. Avg and [iff]] show average and difference of both, which show the bias score L.Ms.
Table I: Experimental results of StereoSet (Nadeem et al. 2020 CrossPairs (Nangia et al. 2020 LMS (language modeling score), SS (stereotype score), and ICAT (idealized CAT score) are bias metrics of StereoSet. All (all dataset), Stereo (stereo dataset), and Anti (anti-stereo dataset) are bias metries of CrowS-Pairg First place written bold.
StereoSet
CrowS-Pairs
Model
Gender
Race
Gender
LMS
SS
ICAT
LMS
SS
ICAT
All
Stereo
Anti.
Llama 3.2
89.9
59.7
72.5
90.1
62.1
68.3
61.9
62.1
60.7
+INLP
90.6
58.3
75.6
90.4
64
65
48.2
48.6
45.9
+SentDebias
91.5
60.2
72.8
90.8
56.3
79.4
51.3
52.2
46
+ADELE
90.7
58.2
75.8
90.5
64.1
65.1
48.3
48.7
45.9
+FaRM
91.4
60.3
72.6
90.8
56.2
79.5
51.4
52.2
46.1
+FineDeb
82.1
54.2
75.2
77
51.8
74.1
57.1
57.1
57.3
+CRISPR
89.7
60.3
72.4
91.0
53.2
81.5
50.8
52.0
47.1
+RB
90.8
54.1
76.7
89.1
54.7
82.8
49.7
52.3
48.3
+Ours
91.6
53.1
86
91.3
54
84
48.1
47.9
49.9
All scores. Model-level debiasing (ADELE, FaRM, FineDeb) and recent deep debiasing modules (HDD) further improve certain metrics the cost of increased computational overhead. Inference- time baselines such as CRISPR offer modest bias reductions without modifying model parameters, but they largely leave the underlying attention representations unchanged.
By contrast, MQAR achieves the best or near-best scores across LMS, SS, ICAT, and CrowS-Pair (All/Stereo/Anti) on LLaMA 3.2. This indicates that quantized attention regularization effectively suppresses stereotypical associations while preserving language modeling quality, outperforming both representationn-leve and inference-time baselines on these benchmarks.
4.3 RESULTS ON DIFFERENT DOWNSTREAM TASKS
This paper next assesses whether MQAR preserves utility while mitigating bias on three downstream tasks: abusive language detection, hate speech detection, and sentiment analysis, well as text
8

---

Published as a conference paper at ICLR 2026
Table 2: Experimental results for downstream tasks. AUC, FPED, and FNED are represented as percentage score. First place written in bold and second place written in underlined.
Model
Task
Type
Metric
Baseline
OSCaR
SentDebais
INLP
Ours
AUC
93.8
93.5
93.6
93.7
93.7
Original
Abusive
FPED
2.32
1.20
2.53
1.92
1.87
Data
language detection (Founta)
FNED
3.71
6.21
3.46
6.34
3.44
AUC
92.3
91.9
92.5
91.5
92.8
Generated
FPED
0.262
0.654
0.131
0.314
0.0654
Data
FNED
0.251
0.036
0.0835
0.332
0.167
AUC
96.5
94.3
96.3
88.2
95.1
Original
FPED
0.121
0.443
0.0117
0.502
0.060
Hate speech detection
Data
FNED
9.54
3.61
4.43
12.4
3.21
AUC
94.7
89.2
94.9
84.8
94.3
(CMSB)
Generated
FPED
0.0584
0.0562
0.0137
0.0192
0.0188
Data
FNED
3.01
1,01
0.0442
0.0257
0.0218
Anger
0.0074
0.0092
0.0121
0.0052
0.0052
Sentiment
Emotion
A1
0.0316
0.0217
0.0149
0.0175
0.0163
analysis (EEC)
Anger
P4
0.0219
0.0159
0.0130
0.0121
0.0133
Valence
A FI-MT
0.0198
0.0137
0.0119
0.0130
0.0105
Text
GPTScore
Bias
8.73
4.36
5.82
4.31
4.21
generation
BLUE
Bias
0.1
0.07
0.11
0.08
0.08
Question
Bias
36.8
33.5
35.3
34.6
35.5
answering
generation setting. For these tasks, this experiment focuses on gender bias and compare GPT- Neo backbone equipped with OSCaR, SentDebias, INLP, and MQAR against the original GPT-Neo baseline. All results are averaged over three random seeds Overall, MQAR achieves an average accuracy drop of at most 0.4 percentage points relative to the non-debiased baseline across all tasks (see Table and Appendix D)
Experiments of abusive language detection are conducted on Twitter datasets from Founta et al. (2018). The experimental results are evaluated on two types of datasets: original dataset and gener- ated dataset. The former shows how bias appears against the original data distribution that is close to the real situation. The latter is constructed with the selection sentences containing female and male phrases from the original datasets and the addition of those replacing female and male phrases, respectively. The generated dataset that gives the distributions of sentences containing female and male phrases has the same bias distribution. FPED and FNED used as bias metries, and AUC is used as accuracy (Park et al. 2018). FPED and FNED are calculated as equation (15) where FPR false positive rate, FNR false negative rate and T is of all groups. The upper side of Table 2 shows the results.
FPED PPR- EITT FNR -
FNR
(15)
OSCaR achieves strong reductions in FPED and FNED but at the cost of a noticeable decrease in AUC, indicating that aggressive projection can remove task-relevant information. SentDebias maintains higher AUC but suffers from information loss due to its projection-basee debiasing. By contrast, MQAR attains AUC that statistically indistinguishable from the haseline while achieving competitive better FPED/ENED scores, showing that it reduce bias with only a small accuracy drop.
Sentiment analysis using BЕC is evaluated based the difference in emotion prediction values between sentences with male phrases and female phrases, where low numbers mean low bias. The anger attribute is identified mainly where gender bias is the most serious. The middle of Table shows its result. The proposed model has lower numerical values than other methods, illustrating that the model effectively removes the bias even for the more difficult downstream tasks.

---

Published as a conference paper at ICLR 2026
In the case of text generation, this study utilizes the bias measurement method proposed by Sun al. This method involves scaling the metrics used to evaluate text generation, and then using the differences in each attribute measure bias levels. Although this method can be applied to various metrics, in this paper, is implemented using BLEU and GPTScore. 'he experimental results are summarized in the lower part of Table 2 Compared with other methods, it is confirmed that the approach presented in this paper is effective in appropriately mitigating bias
4.4 EFFICIENCY AND OVERHEAD
This experiment analyzes the computational overhead introduced by MQAR. Table 3 re- ports the additional parameters, FLOPs per for- ward pass, and end-to-end inference latency for each backbone LLM with different debias- ing methods. On LLaMA 3.2 with StereoSet, MQAR adds only 48. % parameters relative to the backbone and increases FLOPs by 87.0%, yet it achieves approximately 43% lower end- to-end inference time than FineDeb, which re- quires full-model fine-tuning. Similar trends hold for other backbones, where MQAR con- sistently offers favorable trade-off between fairness improvements and computational cost due to its frozen-backbone design and lightweight quantization modules.
Table 3: Computational overhead and latency on Llama 3.2. Param and FLOPs denote relative overhead compared to the frozen backbone (in %), and Latency reports end-to-end inference time per 100 generated tokens (in ms).
Method Backbone FineDeb MQAR
Param
FLOPs
Latency
0.0
0.0
97.8
+107.3
+191.8
189.8
+48.1
+87.0
132.7
4.5 ABLATION STUDY
For the proposed method, this paper checks the results of the absence of the discriminator and decoder to confirm the effect of each compo- nent on bias mitigation. An experiment con- ducted by checking the metric values for abu- sive language detection with the BERT model. Table iillustrates the result.
Table 4: Ablation study on abusive language de- tection. (a) the result of decoder removal and (b) is that of discriminator.
Dataset
Metric
Original
(a)
(b)
AUC
93.9
50.3
93.9
the decoder is removed, the original data can-
not be maintained during bias mitigation, sulting in a significant decrease in accuracy and an increase in the bias metric due to data loss. As a result, it can be confirmed that the de- coder plays an important role in maintaining the original information. If the discriminator is re- moved, the original data retained, but bias mitigation is not done properly, resulting in high bias for the dataset.
Original
FPED
1.84
20.2
2.50
FNED
3.46
18.3
3.46
AUC
92.8
48.9
92.4
Generated
FPED
0.0654
22.1
0.392
FNED
0.167
17.5
0.250
5 CONCLUSION
This paper proposes MQAR, model-agnosti debiasing method that mitigates multi-attribute social bias frozen LLMs via structured quantized regularization. Unlike existing approaches that require fine-tuning or data augmentation, MQAR directly operates self-attention layers, disentangling protected attribute activations while preserving semantic content. Extensive experiments across five LLMs and three standard bias benchmarks show that MQAR achieves consistent bias reduction while maintaining downstream task accuracy within 0.4% of original models.
While MQAR demonstrates strong performance in zero-shot and large-data settings, future work can explore its application in low-resource regimes where robust disentanglement is more challenging. Moreover, extending MQAR to multilingual models and instruction-tun -unned LLMs presents promis- ing directions for broadening fairness-awen awrrre NLP. Overall, MQAR provides a lightweight, scalable solution for fair language modeling, pushing the frontier toward bias-resilient L.L.M deployment.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the Yonsei Fellow Program funded by Lee Youn Jae, IITP grant funded by the Korea government (MSIT) (RS-2022-1122013) Developing a Sustainable Collab- orative Multi-modal Lifelong Learning Framework), and Air Force Defense Research Sciences Pro- gram funded by Air Force Office of Scientific Research.
REFERENCES
Shivaji Alaparthi and Manit Mishra. Bert: A sentiment analysis odyssey. Journal of Marketing Analytics, 92):118-126, 2021.
Zachary Brown, Nathaniel Robinson, David Wingate, and Naney Fulda. Towards neural program- ming interfaces. Advances in Neural Information Processing Systems, 33:17416-17428, 2020.
Somnath Basu Roy Chowdhury and Snigdha Chaturvedi. Learning fair representations via rate- distortion maximization. Transactions of the Association for Computational Linguistics, 10: 1159-1174, 2022.
Xavier Suau Cuadros, Luca Zappella, and Nicholas Apostoloff. Self-conditioning pre-trained lan- guage models. In International Conference Machine Learning, pp. 4455 4473. PMLR, 2022.
Yiwei Dai, Hengrui Gu, Ying Wang, and Xin Wang. Mitigate extrinsic social bias in pre-trained language models via continuous prompts adjustment. Proceedings of the 2024 conference on empirical methods in natural language processing, pp. 11068-11083 2024.
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias bios: A case study of semantic representation bias in high-stakes setting. In proceedings of the Confer- ence on Fairness, Accountabilityy and Transparency, pp. 120-128, 2019.
Sunipa Dev, Tao Li, Jeff M Phillips, and Vivek Srikumar. Oscar: Orthogonal subspace correction and rectification of biases in word embeddings. arXiv preprint arXiv:2007.00049, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04888 04805, 2018.
Zahra Fatemi, Chen Xing, Wenhao Liu, and Caiming Xiong. Improving gender fairness of pre- trained language models without catastrophic forgetting. arXiv preprint arXiv:2110.05367 2021.
Antigoni Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Black- burn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. Large scale crowdsourcing and characterization of twitter abusive behavior. In Twelfth International AAAI Conference on Web and Social Media, 2018
Yue Guo, Yi Yang, and Ahmed Abbasi. Auto-debias: Debiasing masked language models with automated biased prompts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 1012-1023. 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020.
Graeme John Hirst. Anaphora in natural language understanding: A survey. The Australian Na- tional University (Australia), 1979.
Kyle Hsu, William Dorrell, James Whittington, Jiajun Wu, and Chelsea Finn. Disentanglement via latent quantization. Advances in Neural Information Processing Systems, 36, 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Amm Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William Sayed. Mistral 7b, 2023. URL https: wwaxxiv.rrg/abae 06825

---

Published as a conference paper at ICLR 2026
Ting Jiang, Shaohan Huang, Zihan Zhang. Deqing Wang. Fuzhen Zhuang, Furu Wei, Haizhen Huang. Liangjie Zhang, and Qi Zhang. Promptbert: Improving bert sentence embeddings with prompts. arXiv preprint Stth:2221104333. 2022.
Masahiro Kaneko and Danushka Bollegala. Gender preserving debiasing for pre-trained word em- beddings. In Proceedings of the 57th Annual Meeting of the Association for Compulational Lin- guistics, pp. 1641-1650, 2019.
Taeyoun Kim, Jacob Mitchell Springer, Aditi Raghunathan, and Maarten Sap. Mitigating bias in rag: Controlling the embedder. In Findings of the Association for Computational Linguisties: ACL 2025, pp. 18999-190244 2025.
Svetlana Kiritchenko and Saif Mohammad. Examining gender and race bias in two hundred senti- ment analysis systems. In SEM@ NAACL-HLT, 2018.
Anne Lauscher, Tobias Lucken, and Goran Glavaš. Sustainable modular debiasing of language models. Findings of the Association for Computational Linguisties: EMNLP 2021, pp. 4782- 2021.
Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, and Louis- Philippe Morency. Towards debiasing sentence representations. arXiv preprint axXi::2000000008000 2020.
Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. Towards understand- ing and mitigating social biases in language models. In International Conference Machine Learning, pp. 6565-6576. PMLR, 2021.
Yi-Chung Lin and Keh- Yih Su. How fast can bert learn simple natural language inference? In Pro- ceedings of the 16th Conference of the European Chapter of the Association for Computational 2021.
Linguistics: Main Volume, pp.
Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, Lili Wang, and Soroush Vosoughi. Mitigating political bias in language models through reinforced calibration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35(17), pp. 14857-14866, 2021.
Yan Liu, Yu Liu, Xiaokang Chen, Pin-Yu Yu Chen, Daoguang Zan, Min- Yen Kan, and Tsung- Yi Ho. The devil is in the neurons: Interpreting and mitigating social biases in language models. In The twelfth international conference on learning representations, 2024.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey, Adversarial autoencoders. arXiv preprint MxXiv:5511005444 2015.
Marta Marchiori Manerba and Sara Tonelli. Fine- fine-grained fairness analysis of abusive language detection systems with checklist. In Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021), pp. 81-91, 2021.
Justus Mattern, Zhijing Jin, Mrinmaya Sachan, Rada Mihalcea, and Bernhard Schölkopf. Under- standing stereotypes in language models: Towards robust measurement and zero-shot debiasing. arXiv preprint 12 2022.
Chandler May, Alex Wang, Shikha Bordia, Samuel R Bowman, and Rachel Rudinger. On measuring social biases in sentence encoders. arXiv preprint 1903. 2019.
Marzieh Mozafari, Reza Farahbakhsh, and Noël Crespi. Hate speech detection and racial bias miti- gation in social media based on bert model. PloS one, 15(8):c0237861, 2020.
Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.09456 2020.
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel Bowman. Crows-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1953-1967, 2020.
12

---

Published as a conference paper at ICLR 2026
Nafisch Nikeghbal, Amir Hossein Kargaran, and Jana Diesner. Cobia: Constructed conversations can trigger otherwise concealed societal biases in llms. In Proceedings of the 2025 Conference Empirical Methods in Natural Language Processing, pp. 16399-639,,,0 2025.
Jacil Park and Sung Bae Cho. Adversarial discriminator to mitigate gender bias in abusive language detection. In 26th European Conference on Artificial Intelligence ECAI 2023, pp. 1851-1858.
IOS Press BV, 2023.
Jaeil Park and Sung- BBe Cho. Improving faimess of abusive language detection with multi-attribute adversarial latent discriminator. Expert Systems with Applications, pp. 130346, 2025.
Ho Park, Jamin Shin, and Pascale Fung. Reducing gender bias in abusive language detection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018, 2018.
Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. Bbq: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguisties: ACI. 2022, pp. 2086-2105, 2022.
Haoruo Peng, Daniel Khashabi, and Dan Roth. Solving hard coreference problems. In Proceedings of the 2015 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Techmologies, pp. 809-819, 2015.
Reid Pryzant, Richard Diehl Martinez, Nathan Dass, Sadao Kurohashi, Dan Jurafsky, and Diyi Yang. Automatically neutralizing subjective bias text. In Proceedings of the aaai conference on artificial intelligence, volume 34(01), pp. 480-489, 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learming Research, 2 21(1):5485-5511, 2020.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guard- ing protected attributes by iterative nullspace projection. arXiv preprint S00iv::2004.76661 2020.
Akash Saravanan, Dhruv Mullick, Habibur Rahman, and Nidhi Hegde. Finedeb: A debiasing frame- work for language models. arXiv preprint 230.0233,, 2023.
Timo Schick, Sahana Udupa, and Hinrich Schütze. Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in nlp. Transactions of the Association for Computational Linguisties, 114408-1424, 2021.
Patrick Schramowski, Cigdem Turan, Nico Andersen, Constantin A Rothkopf, and Kristian Kerst- ing. Large pre-trained language models contain human-like biases of what is right and wrong to do. Nature Machine Intelligence, 4(3):258 268, 2022.
Seungjae Shin, Kyungwoo Song, JoonHo Jang, Hyemi Kim, Weonyoung Joo. and Il-Chul Moon. Neutralizing gender bias in word embeddings with latent disentanglement and counterfactual gen- eration. In Empirical Methods in Natural Language Processing conference (EMNLP) 2020, pp- 3126-3140. Association for Computational Linguistics, 2020.
Jiaming Song, Pratyusha Kalluri, Aditya Grover, Shengjia Zhao, and Stefano Ermon. Learning controllable fair representations. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 2164-2173 PMLR, 2019.
Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Eliz- abeth Belding, Kai-Wei Chang, and William Yang Wang. Mitigating gender bias in natural lan- guage processing: Literature review. In Proceedings of the 57th Anmual Meeting of the Associ- ation for Computational Linguistics, pp. 1630- 1640, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/PP 1/133/11/P19111// URL https: haap:::://allathooogy..
oorg/P19 1t1///9991115
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. /3971, 2023.
13

---

Published as a conference paper at ICLR 2026
Kellie Webster, Xuezhi Wang. lan Tenney, Alex Beutel, Emily Pitler, Ellie Pavlick, Jilin Chen, Ed Chi, and Slav Petrov. Measuring and reducing gendered correlations in pre-trained models. arXiv preprint arXhv:2010.06032, 2020.
Nakyeong Yang, Taegwan Kang, Stanley Jungkyu Choi, Honglak Lee, and Kyomin Jung. Mitigating biases for instruction-following language models via bias neurons elimination. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 9061-9073, 2024.
Liu Yu, Yuzhou Mao, Jin Wu, and Fan Zhou. Mixup-based unified framework to overcome gender bias resurgence. Pruceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1755-1759) 2023.
Abdelrahman Zayed, Prasanna Parthasarathi, Gonçalo Mordido, Hamid Palangi, Samira Shabanian, and Sarath Chandar. Deep learning on healthy data diet: Finding important examples for fair- ness. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 14593- 14601, 2023.
Dora Zhao, Jerone Andrews, and Alice Xiang. Men also do laundry: Multi-attribute bias amplifica- tion. In International Conference on Machine Learning, pp. 42000-42017. PMLR, 2023.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. arXiv preprint arXiv:1804.0683 2018.
Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tie-Ye Yan Liu. Incorporating bert into neural machine translation. arXiv preprint wrXiv:2002.06222 2020.
14

---

Published as a conference paper at ICLR 2026
Table 5: Parameters for Training BERT, GPT, Llama, T5 and Mixtral
Model
Parameters
Value
Model name EPS
pert-base-uncaser
1.00E-15
latent vector dimension
400
learning rate encoder/decoder
1.00E-04
BERT
learning rate for discriminatoor/lassiieee
5.00E-05
learning rate for ane-ttnnng BERT on downstream task
2.00E-05
random seed
patience for early stop
Model name EPS
1.00E-05
latent vector dimension
1000
learning rate for encoder/decoder learning rate for diseriminator/claasifeer
5,00E-04
GPT
5.00E-05
learning rate for fine-tuning GPT downstream task
1.00E-05
random seed
patience for early stop
Model name
Llama-3.2-0
EPS
1.00E-15
latent vector dimension
800
learning rate encoder/decoder
1.00E-04
Llama
learning rate for discriminnaorr/lassifeer
5.00E-05
learning rate for fine-tuning I.lama on downstream task
2.00E-05
random seed
0
patience for early stop
Model name EPS
15-base
1.00E-06
latent vector dimension
400
learning rate for encoder/decoder
1.00E-04
T5
learning rate for discriminaaorr/llassieer
5.00E-05
learning rate for fine-tuning T5 on downstream task
1. 1.00E-04
random seed
patience for early stop
7
Model name EPS
Mixtral-8e 7B-v0.1
1.00E-06
latent vector dimension
400
learning rate for encoder/decoder
1.00E-04
Mixtral
learning rate for discriminator/classiffi
5,00E-05
learning rate for fine-tuning Mixtral on downstream task random seed
1.00E-04
0
patience for early stop
A PROTECTED-ATTRIBUEE LEXICONS AND SUPERVISION
This appendix describes how the proposed method constructs and validates the weak supervision signals, focusing on lexicons for gender and race, and classifier-bat based labels for other protected attributes such as religion and region.
Gender and race lexicons. For gender and race, this paper follows prior work on fairness in language models and sentimentroxicitybbuuiie classification, which commonly uses demographic term lists weak supervision signals for protected attributes (e.g., gendered pronouns, kinship terms, and occupational titles). Building on these sources, this paper constructs moderate-sine lexicons consisting of gendered and race-related terms that frequently appear in the corpora.
15

---

Published as a conference paper at ICLR 2026
For gender, this paper uses list of masculine and feminine terms covering pronouns. kinship lations, professions, and other common descriptors. The full list used in the experiments is shown below:
Male: he, him, masculine, actor, author, boy, bridegroom, brother, conductor, count, czar, daddy, duke, man, emperor, father, grandfather, heir, host, husband, king. master, murderer, nephew, poet, policeman, prince, sir, son, uncle, wizard, waiter, guy, boyfriend, dad, gentle- man, lord, monk, priest, prophet, patron, viscount, shepherd, steward, baron, peer, abbot, traitor, benefactor, hunter, tempter, enchanter, songster, manservant, landlord, milkman, giant.
Female: she, her, feminine, actress, authoress, girl, bride, sister, conductress, countess, czarina, mummy, duchess, woman, empress, mother, grandmother, heiress, hostess, wife, queen, mistress, murderess, niece, poetess, policewoman, princess, madam, daughter, aunt, witch, waitress, girlfriend, mom, lady, nun, priestess, prophetess, patroness, viscount- ess, shepherdess, stewardess, baroness, peeress, abbess, traitress, benefactress, huntress, temptress, enchantress, songstress, maidservant, landlady, milkmaid, giantess.
For race and ethnicity, this paper constructs lexicon of descriptors (e.g., ethnic and regional iden- tifiers, demonyms) by aggregating terms from prior demographic- benchmarks and public de- mographic resources, followed by manual filtering to remove rare or ambiguous entries. Due to space constraints, this paper does not list all race-related terms here; representative examples and full vocabularies will be released with the code.
Filtering overlaps with target labels. To avoid leakage between protected-attribut supervision and downstream task labels, this paper explicitly removes lexicon entries that coincide with target label tokens. Concretely, for each downstream dataset (e.g., abusive language and hate speech detec- tion), this paper (i) constructs the set of label strings and their tokenization, (ii) automatically drops any lexicon term that appears verbatim in the label vocabulary, and (iii) manually inspects border- line cases where lexicon term could be semantically close label (e.g.. "offensive", "toxic") and removes them if necessary. This procedure ensures that the proposed protected-attribus supervision does not directly encode task-specific labels.
Classifier-based supervision for other attributes. For attributes that are less amenable to stable lexicon construction-suu as religion, region, or other socio-demographic categories-this paper resorts to off-the-shelf sentence-level classifiers. Specifically, this paper applies publicly available pre-trained models (e.g., toxicity/fairness or demographic-attrin treats high-confidence predictions weak labels for the corresponding protected attribute. This paper then binarizes these predictions into multi-hot attribute vectors and uses them in the adversar- ial discriminator. In all cases, this paper only retains predictions above conservative confidence threshold to reduce noise and treats this classifier-based supervision as complementary to lexicon- based labels rather than replacement.
classifiers) to each sentence and
Overall, this hybrid strategy combinigg curated lexicons for gender and race with classifier- based supervision for other attributes, and explicitly filtering out overlaps with downstream la- bels-provee dels-drovides a flexible yet controlled way to obtain protected-attribute signals needed to train MQAR without requiring manual annotation of sensitive attributes.
B HYPERPARAMETERS FOR TRAINING LLMS
The main parameters of the model are summarized in Table and 2. For most hyperparameters, the default values set based on the papers proposing each LLM. The autoencoder of the proposed model is designed as a Multi-Layer Perceptron (MLP) using three layers. Experiments are conducted by adjusting the dimension of each latent vector and the learning rate. Training is incorporated with early stopping, and the related numerical values are also documented in the table. For the downstream tasks, the model has been fine-tuned using each dataset.
16

---

Published as a conference paper at ICLR 2026
Table Experimental results of StereoSet (Nadeem al. 2020 CrossPairs Nangia al. 2020
LMS (language modeling score), SS (stereotype score), and ICAT (idealized CAT score) are bias metric of StereoSet. All (all dataset), Stereo (stereo dataset), and Anti (anti-stereo dataset) are bias metrics of CrowS-Pairs. First place is written in bold.
StereoSet
CrowS-Pairs
Model
Gender
Race
Gender Stereo
LMS
SS
ICAT
LMS
SS
ICAT
All
Anti.
BERT-Bas
85.4
58.3
71.2
88.3
61.7
67.6
60.5
61.1
56.9
BERT- Base+INNP
86.3
57.3
73.7
88.9
63
65.8
48.3
49.7
40
SERT-Base+SentDebias
87.2
59.4
70.8
89.3
55.2
80
51.3
53.2
40.1
BERT-Bas++ADELE
86.3
57.3
73.7
88.9
633
65.8
48.3
49.7
40
BERT-Basc+FaRM
87.2
59.4
70.8
89.3
55.2
80
51.3
53.2
40.1
BERT-BBae++inneDeh
77.7
53.3
72.6
75.4
50.8
74.1
54.6
58.1
51.5
BERT-Base+Ours
87.3
52.3
83.3
89.8
53.1
85.7
48.2
48.9
44.1
T5
84.7
60.2
67.4
88
62.3
66.4
64.2
66.3
51.3
T5+INLP
85.3
58.6
70.6
88.4
64.3
63.1
50.5
52.8
36.4
T5+SentDebias
86.2
60.7
67.7
88.8
56.5
77.3
53.6
56.4
36.6
TS+ADELE
85.4
58.8
70.4
88.4
64.1
63.5
50.4
52.7
36.3
T5+FaRM
86.4
60.8
67.8
88.8
56.3
77.6
53.4
56.3
36.5
T5+FineDeb
76.9
54.7
69.7
74.8
52.1
71.7
59.3
61.3
47.8
T5+Ours
86.4
53.8
79.9
89.2
54.3
81.5
50.4
52
40.6
GPT-Neo
86.1
59.5
70.2
89.1
60.8
71.3
63.0
65.5
53.1
GPT-Neo+INI.
86.6
57.5
74.1
89.2
63.3
66.0
48.7
50.1
40.3
SenDeics
87.4
59.6
71.1
89.6
55.6
80.4
51.5
53.5
40.4
GPT-Nco+ADELE
86.6
57.7
74.0
89.2
63.3
66.1
48.6
49.9
40.2
GPT-Neo+FaRM
87.5
59.7
71.1
89.7
55.4
80.4
51.6
53.5
40.5
GiT-Neo+FinelDeb
78.0
53.6
73.0
75.6
51.1
74.4
54.9
58.4
51.7
GPT-Neo+Ours
87.7
52.5
83.6
90.1
53.4
$6.0
48.4
49.2
44.4
Mixtral
86.9
58.5
72.2
89.9
59.8
73.3
61.8
64.3
52.5
Mixtral+INLP
87.4
56.5
76.1
90.0
62.3
68.0
47.5
48.9
39.7
Mixtral+SSentDebias
88.2
58.6
73.1
90.4
54.6
82.4
50.3
52.3
39.8
Mixtral+ADELE
87.4
56.7
76.0
90.0
62.3
68.1
47.4
48.7
39.6
Mixtral+FaRM
88.3
58.7
73.1
90.5
54.4
82.4
50.4
52.3
39.9
Mixtral+FFineDeb
78.8
52.6
75.0
76.4
50.1
76.4
53.7
57.2
51.1
Mixtral+Ours
88.5
51.5
85.6
90.9
52.4
88.0
47.2
48,0
43.8
C RESULT ON BIAS IN OTHER LLMS
Table shows the result of two datasets, StercoSet (Nadeem et al. 2020) and CrossPairs (Nangia al. 2020), with four LLMs, BERT, T5, GPT-Neo, and Llama 3.2. From these metrics, the ex- perimental results show that, similar as the content of the main paper, INLP, SentDebias, ADELE, FaRM, and FineDeb have mitigated stereotypical associations compared to the original models. The proposed method demonstrated superior performance in these two datasets compared to other al- gorithms, suggesting that it is more effective in preserving semantic elements and removing biased elements in stereotype categories than other algorithms.
D RESULT ON DOWNSTREAM TASKS IN OTHER LLMS
Table shows the result of 4 downstream tasks, abusive language detection (Founta et al. 2018), sentiment analysis (Kiritchenko & Mohammad, 2018), text generation (Sun et al. (2019). question answering (Hendrycks et al. 2020).. with two LLMs, BERT and T5 (the result GPT-Neo is written in section 4.3).
17

---

Published as a conference paper at ICLR 2026
Table 7: Experimental results for downstream tasks. AUC, FPED, and FNED are represented as percentage score. First place written in bold and second place written in underlined.
Model SentDebais
Task
Type
Metric
Baseline
OSCaR
INLP
Ours
BERT
AUC
93.8
93.5
93.7
Original
Abusive language detection (Founta)
FPED
2.32
1.20
2.53
1.92
1.87
Data
FNED
3.71
6,21
3,46
6,34
3.44
AUC
92.3
91.9
92.5
91.5
92.8
Generated
FPED
0.262
0.654
0.131
0.314
0.0654
Data
FNED
0.251
0.036
0.0835
0.332
0.167
AUC
96.5
94.3
96.3
88.2
95.1
Original
FPED
0.121
0.443
0.0117
0.502
0.060
Hate speech detection (CMSB)
Data
FNED
9.54
3.61
4.43
3.21
AUC
94.7
89.2
94.9
84.8
94.3
Generated
FPED
0.0584
0.0562
0.0137
0.0192
0.0188
Data
FNED
3.01
1.01
0.0442
0.0257
0.0218
Anger
APT-M
0.0074
0,0092
0.0121
0.0052
0.0052
Sentiment analysis (EEC)
Emotion
APL-MC
0,0316
0,0217
0.0149
0.0175
0.0163
Anger
APt-M 0.0219
0.0159
0.0130
0.0121
0.0133
Valence
A P
0.0198
0.0137
0.0119
0.0130
0.0105
BERTScore
Bias
8.73
4.36
5.82
4.31
4.21
Text generation
BLUE
Bias
0.1
0.07
0.11
0.08
0.08
AUC
91.4
90.1
89.7
90.7
90.9
Abusive language detection
Original data
FPED
2.45
1.13
2.61
2.03
1.72
FNED
4.03
5.7
3.19
6.66
3.17
AUC
89.4
90.7
89.3
90.1
90.5
(Founta)
Generated data
FPED
0.256
0.605
0.123
0.335
0.071
FNED
0,243
0.038
0.187
0,32
0.183
Sentiment analysis (EEC)
Anger
PL
0.0076
0,0088
0.0132
0.0049
0.0053
emotion
A A
0.0287
0.0214
0.0139
0.0165
0.0117
Anger
A ALT-MU
0.0199
0.0144
0,0132
0.013
0.0121
valence
AFI M
0.0187
0.0134
0.0121
0.0123
0.0107
GPTScore
Bias
5.51
4.12
4,48
4.36
4.01
Text generation
BLUE
Bias
0.09
0.07
0.1
0.09
0.08
Question answering
Bias
29.1
26.9
24
24.3
28.8
E RESULTS ON BIAS IN LLAMA WITH STANDARD DEVIATION
In the main paper (Table D. this paper reported bias results on StereoSet and CrowS-Pairs for LLaMA 3.2 and several debiasing methods using single-point estimates. For completeness, this section provides the corresponding results with mean + standard deviation over multiple random seeds, restricted to the LLaMA 3.2 backbone. This paper splits the results by dataset for readability: Table StereoSet scores, and Table 9 reports CrowS-Pain scores.
Across both benchmarks, this paper observes that the variance across runs is relatively small for all methods, indicating that the reported improvements are stable rather than artifacts of a single seed. On StereoSet. MQAR achieves the best or near-best ICAT scores while substantially reducing the stereotype score (SS) for both gender and race, without degrading the language modeling score (LMS). On CrowS-Pairs, MQAR attains the lowest (or competitive) bias scores on the All and Stereo subsets while keeping performance patterns are consistent with the main-tabie findings and support the claim that MQAR provides robust fairness gains LLaMA 3.2 with minimal variance across
the Anti subset comparable to other methods. These
18

---

Published as a conference paper at ICLR 2026
Table 8: StereoSet results for LLaMA 3.2 with debiasing methods, reported mean + standard deviation over multiple runs. LMS, SS, and ICAT are language modeling, stereotype, and idealized CAT scores, respectively, Best values are bold.
Gender SS
Race
Model
LMS
ICAT
LMS
SS
ICAT
LLaMA 3.2
89.9:0.22
59.7+:0.4
0..5
90.1+0.2
62.140..4
08.3
+INLP
90.6:t0.2
58.320.3
75.6=004
90.4+0.2
64.0::0.3
65.0=10,4
+SentDebias
91.510..3
60.210.6
72.8+0.77
90.8+10.3
56.3:10.6
79.4=0.7
+ADELE
90.7+t0.3
58.2+:0.5
75.8++0.6
90.5+0.3
64.1++0.5
65.1+006
+FaRM
91.4+t0.3
60.3+0.6
72.6+0.6
90.8+0.33
56.2++0.6
79.5+0.6
+FineDeb
82.110.6
54.2::0.8
75.2.00.9
77.010.7
51.8
74.1
+CRISPR
89.7+10.4
60.3
72.4+0.7
91.0+0.4
53.240.6
81.5+0.77
+RB
90.8++0.2
54.11:0.3
76.7+10.4
89.1+0.2
54.7.+0.3
82.8+0.44
+MQAR
91.6++0.2
53.140.3
86.0+0.4
91.3+0.2
54.0+F0.3
84.0+0.4
Table 9: CrowS-Pain, results for LLaMA 3.2 with debiasing methods, reported mean + standard deviation over multiple runs. All, Stereo, and Anti denote the overall, stereotype, and anti-stereotype subsets, respectively. Best values are in bold.
Model
All
Stereo
Anti
LLaMA 3.2
61.910.6
62.110.6
60.7
+INLP
48.2+0.5
48.6+0.5
45.9=0.5
+SentDebias
51.3=0.88
52.2+0.9
46.0
+ADELE
48.3++0.6
48.7+0.6
45.9+-0.6
+FaRM
51.4++0.7
52.2+0.7
46.1+.0.7
+FineDeb
57.1
57.1#1.2
57.3+1.0
+CRISPR
50.8+0.9
52.0+0.9
47.1+0099
+RB
49.740.5
52.3+0.5
48.3
+MQAR
48.1100.4
47.9.0.4
49.9
EFFECTIVENESS OF STRUCTURED QUANTIZATION
To isolate the effect of quantization in the proposed MQAR method, this section conducts an ablation study comparing three variants:
Full MQAR: with both vector quantization and adversarial regularization.
No-Quant MQAR: with adversarial regularization but without quantization.
Quant-only: applying quantization without adversarial regularization.
This paper evaluates these variants on the StereoSet dataset using LLaMA 3.2. As shown in Ta- ble I0 Full MQAR achieves the lowest bias scores (SS) and the highest ICAT while slightly im- proving LMS over the baseline. Removing quantization (No-Quant MQAR) weakens attribute dis- entanglement and yields substantially worse ICAT, especially in scenarios with multiple protected attributes. Conversely, using quantization alone (Quans-only) reduces bias only marginally and even harms ICAT, indicating that the quantized bottleneck requires the adversarial regularizer to properly separate protected-atee wroteceed-attribute information from task-relevant semantics.
This section also studies the impact of the quantization codebook size K in the latent space. In practice, this paper selects K from a small candidate set (K E (2,3, 4, 8}) based on validation performance and keep K fixed across layers for simplicity. The sensitivity analysis shows that MQAR is robust to moderate changes in K: very small codebooks (e.g., K = 2) overly compress the latent space and slightly hurt LMS, whereas very large codebooks weaken the debiasing bottleneck and increase SS. A moderate choice such as = consistently yields the best fairness- utility trade-off across benchmarks, and is therefore used in all main experiments.
19

---

Published as a conference paper at ICLR 2026
Table 10: Effect of quantization on bias mitigation for LLaMA 3.2 on StereoSet (gender). LMS, SS,
and ICAT denote language modeling, stereotype, and idealized CAT scores, respectively.
Method
LMS
SS
ICAT
Full MQAR (ours)
91.6
53.1
86.0
No-Quant MQAR
90.7
55.1
67.8
Quant-only
90.5
58.3
63.9
Baseline (no mitigation)
89.9
59.7
72.5
K 2
90.9
52.5
82.1
K 3
91.6
53.1
86.0
K 4
91.5
54.0
84.7
K 8
91.4
55.6
80.3
G OMITTED PROOF FOR LEMMAS AND THEOREM
Lemma S. For any conditional distribution p.
I (R; (R) E. ( (om))))))))))e [1 + H (x).o)
(16)
where D2 is the function of decoder 2 d2, q is the encoder function, p is the real distribution, is the projection function, finctionn and is the entropy.
Proof. As information recovered with Decoder 2:
I [R; = E ( [E [ sp(rass)))) [10gp((()))))]]]] + H((a))
(17)
2 E= m(rlz) [Bay [Bal mpaapl)))
(1 H(ras)
(18)
E و - (mma))))mttl))))t ( + H((go))
(19)
Lemma 6. For any conditional distribution m,
(R: S / (rlz)p()) log m(rlz) s(r)
= C1
(20)
Proof.
Using additional function m, information is transformed from z to and it is recovered with
D2-
(R; A) 1 D2 (m (R))) $1 (R))
(21)
(22)
$ / (rlz) log (rl(2) s(r)
(23)
Corollary 7. If DKL((((()) h (alr)) 1
[(R; A) $ Ep(r.a) [log (a 1r) - (a)
(24)
Proof. For the tighter upper bound when assuming optimal dual regularization, I(R; A) can be written shown in Equations (5)). where D is KL divergence and h is any distribution.
7 (R; A) (KIR.A)DII(Pl r) M(a)) (x || h(a))
(25)
20

---

Published as a conference paper at ICLR 2026
Where DEL 2 0, it shows:
I (R; PRill)) (r) h(a)) (xz(p(a) || h(a))
(26)
Note that the distribution of is traceable which implies the Equations 27 and
Ep(raa [log (a r) - logp(aa)
(27)
A) (Dzz h(a)) - DxL(p(arr) |I (alr))
(28)
2 E r) h(a)) -1
(29)
where is positive such that DRI(p(a(r) (a(r)) < L. From Equation (8) and (11),
(R; & Ep(R,A) plogp(a (r) logg (a)] +
(30)
H DETAILED INFORMATION ON DATASETS
The WinoBias dataset is the benchmark for paired male and female cross-reference-solving exam- ples according to the Winograd format (Hirst, 1979 Peng et al.) 2015). It contains two different subsets: the pro-stereotype where pronouns are utti-steressyps primarily related to the gender- related occupation of pronouns, and the anti-stereotype when the opposite relationship is true. Each subset consists of two types of sentences. One a sentence that requires a semantic understanding of the sentence for cross-reference resolution (Semantics Only), and the other is a sentence that relies on syntactic cues (w/ Semantic Cues). Gender bias is measured using the performance difference between typical subset and a semi-structured subset.
StereoSet dataset a benchmark dataset that measures the social bias of LLMs that model. Gen- der and race are selected among the domains of it. LMS (language modeling score), SS (stereotype score), and ICAT (idealized CAT score) are bias metrics for StereoSet. The closer to 100 on LMS and ICAT, and the closer 50 on SS, the more the bias is ideally mitigated. LMS is value for the meaningful association, SS is a value for the stereotypic association, and ICAT represents a comprehensive number of these two figures. The CrowS-Pair dataset is designed evaluate social biases in textual models, containing pairs of sentences that highlight biases and stereotypes, in the Cross-Pairs-stereo and Sroow--airs-aattsteeee categories. The dataset allows for the measurement of biases by comparing the model's 'ss responses to both biased and unbiased statements in the pairs.
1 TABLE OF RELATED WORKS
Mitigating bias in large language models (LLMMs) has received substantial attention, with prior works proposing methods across different stages of model deployment. Based on Table this paper sum- marizes these approaches along three dimensions: model-level interventions, sentence-level repre- sentation manipulation, and prompt-based inference-time strategies.
Model-level Approaches. These methods directly modify the training process of LLMs. Projection-based techniques, such as SentDebias (Liang al.) 2020) and OSCaR (Dev et al. 2020), remove protected attribute components by projecting representations onto orthogonal sub- spaces. While simple, such methods risk reducing representation capacity for downstream tasks (Shin et al. 2020). Regularization-bant based methods, including FaRM (Chowdhury & Chaturvedi 2022) and dropout- based debiasing (Webster et al. 2020), introduce fairness constraints into train- ing losses. Others exploit token-level statisties, such as the masked token divergence in (Guo al., 2022]. However, all these methods typically require access to training data and full model fine- tuning, limiting scalability new domains closed-source I.LMs.
21

---

Published as a conference paper at ICLR 2026
Table 11: Related works for reducing bias of large language models (LLMs).
Target
Method
Description
Hard debiasing on layer result (Liang al. 2020)
Projection based
Projection on orthogonal subspace with correction and rectification (Dev al. 2020)
Drop out regularization (Webster et al. 2020)
Post-hoc debiasing procedure using
Regularization based
intrinsic data (Schick et al. 2021)
Learning Model
Debiasing using the rate-distortion function
(Chowdhury & Chaturvedi 20222)
[MASK] token based
Cansen-Shannon divergence with mask probability of pair sentences with protected feature (Guo al.
2022)
Generative self-condin self-conditioning methods (Cuadros al.
2022)
Sustainable modular debiasing based on dedicated debiasing
Data based
adapters with counterfactual data (Lauscher al.
2021
Combining important counterfactual Zayed et al.
2023
Projection based
Interactive null space projection (Ravfogel et al. 2020
Embedded Sentence
Autoregressive interactive projection (Liang et al 2021
Regularization based
Reinforced calibration on embedding (Luu al. 202
LSTM-baased bias mitigation on BERT (Pryzant et al. 2020)
Analyzing the bias mitigation effects of prompts in various levels of abstraction (Mattern et al. 2022)
Static-prone prompt
Prompt of LLM
Dynamic- prompt
Using gender-neutral datasets for prompt update
biased word embeddings (Fatemi et al. 2021
Embedding-level Techniques. These operate the latent sentence representations of pre-trained models. INLP (Ravfogel et al.) (2020) and its variants project sentence embeddings away from attribute-relevant directions. More recent work combines LSTM-based debiasing with BERT (Pryzant et al., (020). However, such techniques may over-constrain the latent space, leading to information loss degraded performance.
Prompt-based Inference-time Strategies. These methods aim mitigate bias without modifying model parameters, using static or dynamic prompts. Static prompt tuning (e.g., (Mattern et al, 2022 adjusts prompt phrasing proposed by Fatemi et al. (2021). retrain embeddings with gender-neutral prompts or counterfactual data. While appealing due to their low-cost deployment, these methods cannot correct internal representational biases embedded within attention layers.
reduce bias exposure. Dynamic prompt updates, such as those
Gap in Multi-Attribute Mitigation. Despite progress, most existing methods target single- attribute settings (e.g., gender only) and fail to generalize across intersecting attributes. Moreover, techniques that modify embeddings or prompts fail to address internal bias entanglement at the self-attention level-a known amplifier of representational bias (Jiang et al. 2022).
Our Contribution. MQAR addresses these gaps by introducing quantized regularization frame- work that operates directly on frozen self-attention layers. Unlike mojection-based prompt-based methods, MQAR disentangles and suppresses protected attribute signals within attention outputs, without requiring model fine-tuning. To our knowledge, it is the first scalable, multi-attribute debi- asing method that is model-agnostic and operates entirely post-hoc, making it suitable for black-box LLM deployments.
22


Reviews:
[{"review_id": "oiVQnpcfwH", "reviewer": "Reviewer_DGTG", "summary": "This paper addresses the critical issue of multi-attribute social bias such as race and gender in Large Language Models, noting that existing debiasing methods are often limited to single attributes, require expensive fine-tuning, or fail to generalize across different architectures. The authors introduce Multi-feature Quantized Attention Regularization or MQAR, a novel plug-in method that operates on the frozen self-attention layers of pre-trained LLMs. This method uses structured quantization to create a discrete bottleneck that disentangles attribute-specific activations, combined with a discriminator-guided autoencoding regularizer to adversarially suppress the identified bias while preserving the original semantic representations. MQAR was evaluated on five diverse LLMs including BERT, T5, GPT-Neo, Mixtral, and LLaMA 3.2, using the WinoBias, StereoSet, and CrowS-Pairs benchmarks. The results show that MQAR consistently outperformed prior methods in bias mitigation while maintaining downstream task accuracy within 0.4 percent of the non-debiased baselines.", "questions": "* There is a mismatch between the paper's multi-attribute motivation and its single-attribute evaluation. It is unclear why a true intersectional benchmark like BBQ was not used.\n* The paper does not justify its exclusion of 2024-2025 SOTA inference-time methods from its performance comparison.\n* The paper does not quantify its \"lightweight\" claim. The actual increase in parameters and inference latency from adding the new modules is missing.\n* The abstract's claim of a 0.4 percent accuracy preservation needs clarification.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The method introduces a novel use of structured quantization, typically for compression, as a fairness bottleneck to achieve disentanglement.\n* The method is a model-agnostic plug-in that works on frozen LLM backbones, which avoids expensive fine-tuning and makes it broadly applicable.\n* The method demonstrates strong empirical results, consistently outperforming chosen baselines on bias metrics across all five tested LLMs.\n* A strong ablation study clearly validates the design, proving the decoder is crucial for utility and the discriminator is crucial for bias reduction.", "weaknesses": "* The state-of-the-art comparison is critically outdated, as it ignores modern inference-time methods and only compares against techniques from 2020-2022.\n* The \"lightweight\" claim is unproven, lacking any quantitative analysis of the parameter increase or inference latency overhead, which could be significant from adding modules to every attention layer.\n* The paper's central claim of handling \"multi-attribute\" bias is unsubstantiated, as the evaluation only tests single attributes separately, not intersectional bias, and fails to use appropriate benchmarks like BBQ.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FWQALr6z4k", "reviewer": "Reviewer_NqdJ", "summary": "This paper proposes Multi-feature Quantized Attention Regularization (MQAR) — a lightweight, architecture-agnostic framework for mitigating multi-attribute bias in large language models (LLMs). Instead of retraining or fine-tuning, MQAR intervenes inside frozen self-attention layers by introducing a structured quantization module that disentangles protected-attribute activations (e.g., gender, race) from semantic information. The paper uses an adversarial autoencoder to regularize the latent space to minimize mutual information between latent vectors and protected attributes while preserving task-relevant content. And later in the experiment, the author reported results on five LLMs over three bias benchmarks. The framework also generalizes across tasks such as abusive-language detection, sentiment analysis, and text generation.", "questions": "* Could the author clarify whether these features were intersectional evaluation or independently debiased in separate runs? If independent, how does MQAR handle correlated protected attributes during training?\n* Have you evaluated MQAR on non-English datasets or cross-domain text to confirm this generalization?\n* The quantization process introduces a codebook of K latent vectors. How is the K chosen and is it fixed globally?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "* This paper is structured and well-written. Notations are internally coherent.\n* Hyperparameters and training details are clearly documented in the Appendix\n* The paper result is strongly supported by the reported experiments on multiple LLMs and benchmarks.\n* The quantized regularization integrates discrete latent bottlenecks with adversarial training to disentangle sensitive features.", "weaknesses": "* In the evaluation of multi-attribute bias, metrics are aggregated by gender and race, but intersectional sub-groups are not explicitly analyzed (such as gender & race).\n* The theoretical part is not empirically validated. The fairness claims rely primarily on benchmark reductions rather than statistical significance testing.\n* In Appendix E, the author shows that the Full version yields the lowest bias scores and best fairness–utility trade-off. It remains unclear how quantization changes what the attention representations look like.\n* Improvements are reported as point differences; the significance or variance analysis remains unclear.\n* The paper claims MQAR is architecture- and domain-agnostic. Since all evaluation datasets are English-only, it remains unclear whether MQAR would work equally well on other languages to support the claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h5DRUOB3kV", "reviewer": "Reviewer_JGSk", "summary": "This paper proposes a novel debiasing method called MQAR (Multi-feature Quantized Attention Regularization). MQAR acts as a \"plug-in\" module that injects structured quantization and a discriminator-guided autoencoding regularizer into the frozen self-attention layers of LLM.\nThe goal is to disentangle activations related to sensitive attributes (like race and gender) while preserving core semantic information, all without modifying the LLM's backbone parameters.", "questions": "1. How were the protected-attribute lexicons constructed and validated? Have you evaluated whether these lexicons introduce spurious correlations with downstream labels?\n2. Could you report mean ± standard deviation over multiple random seeds for both fairness and utility metrics, to confirm that improvements are consistent rather than incidental?\n3. Do you have or plan to include explicit experiments on intersectional categories (e.g., Black women vs. White men) to substantiate the “multi-feature” claim?\n4. Please provide quantitative data on the computational overhead (training time, memory, or latency) introduced by MQAR.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality - The work is notable for introducing a modular, architecture-agnostic fairness mechanism that integrates quantization and adversarial autoencoding within frozen attention layers -- an uncommon yet elegant direction compared to embedding- or output-level debiasing. The explicit quantization-based feature disentanglement contributes a novel angle to fairness research in LLMs.\n\nClarity - The paper is written clearly, with an intuitive explanation of how quantization can suppress protected-attribute representations. Figures illustrating the architecture improve readability and understanding.", "weaknesses": "1. The fairness regularization relies partly on lexicon-based weak supervision for protected attribute identification. However, details on lexicon construction, coverage, and potential leakage (e.g., words correlated with downstream labels) are insufficient. This could introduce hidden bias, undermining claims of debiasing generality.\n2. Although the abstract highlights multi-attribute bias, most empirical results focus on single attributes (e.g., gender, race). A clearer demonstration of intersectional fairness (e.g., gender × race) would strengthen the paper’s central claim.\n3. While the paper emphasizes low overhead, quantitative results on latency or memory cost are missing. Such measurements would enhance the credibility of the lightweight claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zzTlx1Ghqv", "reviewer": "Reviewer_kQ3w", "summary": "This paper proposes MQAR to solve the problem of social bias across multiple sensitive attributes in LLMs by injecting structured quantization into frozen self-attention layers. The idea is that MQAR disentangles attribute-specific activations and removes biased information through adversarial autoencoding.", "questions": "- The paper provides an anonymous GitHub link for code release, but why the repository currently contains no implementation or documentation?\n\n- Many recent LLMs undergo alignment or instruction tuning, which already mitigates social bias to some extent. Could you clarify whether the proposed method still provides meaningful improvements on such aligned models, and provide empirical evidence if possible?\n\n- What kinds of bias (semantic vs. syntactic) are most affected by applying the proposed method?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This paper tackles the important and underexplored issue of multi-attribute bias in LLMs, which reflects real-world fairness challenges beyond single-attribute settings (e.g., only focus on gender or race in previous work).\n\n- The proposed framework is architecture independent and focuses on frozen self-attention layers without fine-tuning, which is applicable in practice.", "weaknesses": "- My major concern is that the paper claims that LLMs still exhibit strong social biases, but does not provide empirical evidence that modern instruction-tuned or RLHF-aligned models suffer from such bias to a meaningful extent. Most of the reported results are on static benchmarks and may not reflect the behavior of current-generation aligned models in realistic contexts. Although they evaluated MQAR on five models, none of these were instruction-tuned or alignment-optimized versions, so the claim about persistent social bias in modern LLMs is not strongly substantiated by the experiments as they only showed bias in pre-trained and non-aligned models. This makes the empirical motivation of the paper less persuasive. Also, the authors provided an anonymous GitHub repository link, no code is actually available, which raises concerns about the reproducibility of the experiments and the credibility of the reported results.\n\n\n- The claim that \"semantic content is preserved\" relies solely on performance metrics (claiming a minimal 0.4% accuracy loss in downstream tasks), which may not fully capture nuanced information degradation in the embeddings. Quantization typically introduces discrete bottlenecks that compress information, and while the authors include a commitment loss and reconstruction loss to mitigate this (Eq. 4–6), they never directly analyze trade-offs between bias reduction and information retention. The theorems and lemmas in Section 3.2 primarily focus on establishing the optimization framework for quantized regularization and adversarial training, such as bounding mutual information $ I(R; X) $ and $ I(R; A) $ rather than addressing the trade-off or the specific changes in mutual information before and after debiasing. While the lemmas provide theoretical bounds for mutual information to guide debiasing, the paper lacks empirical validation of these changes across the evaluated datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)