Title: Multi-Feature Quantized Self-Attention for Fair Large Language Models
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Large language models, multi-attribute social bias, quantized adversarial autoencoder

Abstract: Large Language Models (LLMs) often encode social biases tied to sensitive features such as race and gender, undermining fairness in downstream tasks. Existing debiasing methods often fail to generalize across diverse LLM architectures and neglect attention-derived representations, leading to compromised task performance. Moreover, most approaches are tailored to single-attribute settings, limiting their effectiveness under multi-attribute scenarios. This paper introduces Multi-feature Quantized Attention Regularization (MQAR), a novel method for mitigating multi-attribute bias by injecting structured quantization into frozen self-attention layers. MQAR disentangles attribute-specific activations using quantized regularization, while preserving semantic representations through a discriminator-guided autoencoding regularizer. Crucially, the proposed method operates without modifying the backbone parameters of the LLM, ensuring architecture-agnostic applicability and minimizing representation distortion. This paper evaluates MQAR on five diverse LLMs (BERT, T5, GPT-Neo, Mixtral, LLaMA 3.2) using three standard bias benchmarks (WinoBias, StereoSet, CrowS-Pairs). The results show that MQAR consistently outperforms prior methods in mitigating bias in multiple attributes, while maintaining downstream accuracy within 0.4\% of non-debiased baselines. Furthermore, MQAR demonstrates robustness across sentiment analysis, abusive language detection, and text generation tasks. These findings highlight quantized attention regularization as a scalable and effective solution for bias mitigation in modern language models. Code is available on https://anonymous.4open.science/r/MQAR-8C25/.