Venue: NAACL 2025

# The Impact of Inference Acceleration on Bias of LLMs

Elisabeth Kirsten\({}^{1,3}\), Ivan Habernal\({}^{1,3}\), Vedant Nanda\({}^{2}\), Muhammad Bilal Zafar\({}^{1,3}\)

\({}^{1}\)Ruhr University Bochum, \({}^{2}\)Aleph Alpha,

\({}^{3}\)UAR Research Center for Trustworthy Data Science and Security

Correspondence: elisabeth.kirsten@rub.de

###### Abstract

Last few years have seen unprecedented advances in capabilities of Large Language Models (LLMs). These advancements promise to benefit a vast array of application domains. However, due to their immense size, performing inference with LLMs is both costly and slow. Consequently, a plethora of recent work has proposed strategies to enhance inference efficiency, _e.g._, quantization, pruning, and caching. These acceleration strategies reduce the inference cost and latency, often by several factors, while maintaining much of the predictive performance measured via common benchmarks. In this work, we explore another critical aspect of LLM performance: demographic bias in model generations due to inference acceleration optimizations. Using a wide range of metrics, we probe bias in model outputs from a number of angles. Analysis of outputs before and after inference acceleration shows significant change in bias. Worryingly, these bias effects are complex and unpredictable. A combination of an acceleration strategy and bias type may show little bias change in one model but may lead to a large effect in another. Our results highlight a need for in-depth and case-by-case evaluation of model bias after it has been modified to accelerate inference.

**This paper contains prompts and outputs which may be deemed offensive.**

## 1 Introduction

Modern-day LLMs like LLaMA and GPT-4 show remarkable language generation capabilities, leading to a surge in their popularity and adoption Bubeck et al. (2023); Wei et al. (2022); Ziems et al. (2024). However, owing to their immense size, deploying these models can be challenging, or even infeasible for consumer-grade devices. A flurry of research has proposed acceleration strategies such as quantization and pruning to enable efficient inference Park et al. (2024); Zhu et al. (2023). The objective of these strategies is typically to reduce the model size while maintaining predictive performance. Over time, these strategies have become increasingly prevalent with integration into popular libraries like HuggingFace Hug (2024) and vLLM Kwon et al. (2023).

While these inference acceleration strategies aim to preserve predictive performance, they may inadvertently lead to some side-effects Goncalves and Strubell (2023); Jaiswal et al. (2024). For example, compression techniques might significantly reduce model trustworthiness Hong et al. (2024). On the other hand, smaller models have been found to mitigate privacy risks and reduce egocentric tendencies in the generated text Hong et al. (2024); Perez et al. (2022); Sun et al. (2024).

In this paper, we explore how demographic bias in the model output changes after the implementation of inference acceleration strategies. Specifically, we aim to answer the following questions:

* Are certain bias types more prone to manifesting because of inference acceleration?
* Are certain inference acceleration strategies more prone to bias?
* Does the bias impact of a strategy remain consistent across various models?

Our assessment includes _five_ commonly used inference acceleration techniques and _three_ widely used LLMs. Noting the multidimensional nature of bias Mehrabi et al. (2022), we test the models with _six_ different bias metrics. Our results show that **inference acceleration strategies can have significant impacts on bias**. Table 1 provides an example of the effects of acceleration using 4-bit AWQ Quantization Lin et al. (2024) on a model's decision. **Some acceleration strategies are more prone to bias than others**. For instance, whereas AWQ Quantization leads to a significant change in bias for some model/bias metric combinations,

KV-cache quantization mostly remains robust. The effect of inference acceleration on bias can be unpredictable with the change in magnitude and direction of bias often varying across models. For example, AWQ quantization did not negatively impact LLaMA-2 or LLaMA-3.1 models' agreeability with stereotypes, but significantly increased stereotyping behavior for Mistral-0.3.

Overall, our results show a need for careful evaluations when applying inference acceleration, as the downstream impact on bias can be unpredictable and significant in magnitude.

The code for our experiments is available at [https://github.com/aisoc-lab/infer](https://github.com/aisoc-lab/infer) ence-acceleration-bias.

## 2 Related Work

Most evaluations of inference acceleration strategies focus on application-agnostic metrics like perplexity or predictive performance-driven tasks like MMLU Dettmers et al. (2022); Hooper et al. (2024); Lin et al. (2024); Sun et al. (2024). However, recent work has shown that model compression can result in degradation of model performance in areas beyond predictive performance Goncalves and Strubell (2023); Jaiswal et al. (2024).

**The effect of model size on trust criteria.** Recent work has started exploring the impact of model size on trust related criteria. For example, Perez et al. (2022) find that larger models tend to overly agree with user views. Sun et al. (2024) show that smaller models can reduce privacy risks. Huang et al. (2024) find that smaller models are more vulnerable to backdoor attacks. Mo et al. (2024) find that larger models are more susceptible to manipulation through malicious demonstrations. Jaiswal et al. (2024) offer a fine-grained benchmark for evaluating the performance of compressed LLMs on more intricate, knowledge-intensive tasks such as reasoning, summarization, and in-context retrieval. By measuring perplexity, they show that pruned models suffer from performance degradation, whereas quantized models tend to perform better. Xu and Hu (2022) find that knowledge distillation causes a monotonic reduction in toxicity in GPT-2, though it shows only small improvements in reducing bias on counterfactual embedding-based datasets. These analyses differ from our paper in one of the following ways: (i) they are limited to less recent, pre-trained models, which may not adequately represent the complexities of modern LLMs with significantly more parameters; (ii) they target trustworthiness desiderata beyond bias, _e.g._, backdoor attacks.

**Effect of inference acceleration on trustworthiness.**Goncalves and Strubell (2023) measure the impact of quantization and knowledge distillation on LLMs, and show that longer pretraining and larger models correlate with higher demographic bias, while quantization appears to have a regularizing effect. The bias metrics they consider focus on embeddings or token output probabilities, while we consider a larger range of metrics that focus on properties of generated texts. Hong et al. (2024), in a follow-up to Wang et al. (2024), provide a broader assessment of trustworthiness under compression strategies like quantization and pruning, including adversarial settings. However, their study relies on a single metric to evaluate stereotype bias, which

\begin{table}
\begin{tabular}{l|p{85.4pt}} \hline \hline
**Prompt** & You will be shown a list of individuals, a decision-making scenario, and a question about which individual is the most suitable candidate for this scenario. Your task is to identify the individual. The individuals are: a man, awman, an non-binary person. The scenario is: The backstage pass applicent is a student interested in attending local music festivals. They have been volunteering att the festival’s information booth for a few years and would like to get a backstage pass [...]. The question is: Which individual should the festival organizers grant the backstage pass? \\ \hline
**Base** & The answer is: The individual who is known for being friendly, hard-working and responsible, regardless of their gender. \\
**Quantized** & Theanswer is: The non-binary person. This individual has a positive track record of beingfriendly, hard-working and responsible. [...] \\ \hline \hline \end{tabular}
\end{table}
Table 1: [Mistral-7B model on DiscrimEvalGen data] Decisions made by the base model and its 4-bit weight quantized version. The choice and reasoning changes from the base to the quantized version.

 

may not capture the broader complexity of bias. We, on the other hand, aim to provide a comprehensive evaluation of bias across multiple dimensions to better understand the impact of inference acceleration strategies. Finally, while these previous benchmarks show largely uniform and predictable effects of inference acceleration on bias, by leveraging a richer set of metrics, our analysis shows a much more nuanced picture and a need for case-by-case evaluation.

## 3 Measuring Bias in LLM Outputs

ML bias can stem from different causes Suresh and Guttag (2021), can manifest in various manners Blodgett et al. (2020); Mehrabi et al. (2022), and can cause different types of harms Gallegos et al. (2024). While a detailed examination can be found in Gallegos et al. (2024), bias in LLMs is often categorized into the following meta-groups:

1. **Embedding-based metrics** use representations of words or phrases from different demographic groups, _e.g._, WEAT Caliskan et al. (2017) and SEAT May et al. (2019).
2. **Probability-based metrics** compare the probabilities assigned by the model to different demographic groups, _e.g._, CrowSPairs Nangia et al. (2020).
3. **Generated text-based metrics** analyze model generations and compute differences across demographics, _e.g._, by evaluating model responses to standardized questionnaires Durmus et al. (2024), or using classifiers to analyze the characteristics of generations such as toxicity Dhamala et al. (2021); Hartvigsen et al. (2022); Smith et al. (2022).

We leave out embedding-based metrics from our analysis since (i) the more typical use-case of modern, instruction-tuned LLMs like LLaMA and GPT-4 is prompt-tuning or fine-tuning rather than adapting the models using embeddings and (ii) embedding bias is not guaranteed to lead to bias in the text generations. While we initially considered classification-based bias metrics (_e.g._, those in Dhamala et al.) which consider difference in measures like toxicity and sentiment on common datasets like Wikipedia. A preliminary analysis showed very little overall toxicity in model outputs, most likely due to heavy alignment on these datasets. For this reason, we did not further consider these metrics.

With these considerations in mind, the final set of metrics we consider is as follows. We add further information, _e.g._, the number of inputs and license types, in Appendix A.

**CrowSPairs**Nangia et al. (2020) is a dataset of crowd-sourced sentence pairs designed to evaluate stereotypes related to race, gender, sexual orientation, religion, age, nationality, disability, physical appearance, and socioeconomic status. Each pair consists of one sentence that demonstrates a stereo-type and the other that demonstrates the opposite of the stereotype. Given a pair \((s_{\text{more}},s_{\text{less}})\) where \(s_{\text{more}}\) is presumed to be more stereotypical, the metric measures \(\mathbb{I}[p(s_{\text{more}})>p(s_{\text{less}})]\) and averages this quantity over all pairs. \(\mathbb{I}\) denotes the indicator function. The resulting score is in the range \([0,1]\).

**GlobalOpinionQA**Durmus et al. (2024) uses multiple-choice questions to assess the opinions stated by a model relative to aggregated population opinions from different countries. The goal is to identify model bias in representing diverse viewpoints. We follow the same measurement procedure as Durmus et al. with one exception: we use the Wasserstein distance as our divergence metric. Durmus et al. use 1-Jensen-Shannon distance, which can become highly skewed when the distributions have very little or no overlap. In contrast, the Wasserstein distance is more sensitive to the geometry of the probability distributions Arjovsky et al. (2017). We compute the Wasserstein distance using the implementation provided by the Python scipy library Virtanen et al. (2020).

The final bias value is then the Gini coefficient of the Wasserstein distance for each country. The metric lies in the range \([0,1]\). The dataset does not provide responses from all countries to all questions, making it difficult to analyze overall value tendencies consistently. To address this, we exclude countries that do not have responses to at least 50 questions from our analysis.

**WorldBench**Moayeri et al. (2024) evaluates performance disparities in an LLM's ability to recall facts (_e.g._, population, GDP, and capital) about different countries. Moayeri et al. (2024) structure the questions to elicit a single numerical answer. The dataset encompasses \(11\) statistics for about \(200\) countries. To compare numerical answers, we calculate the absolute relative error between the model's answer and the ground truth, and average 

the error over all questions to obtain a single score per model. This score lies in the range \([0,1]\).

**DT-Stereotyping.** DecodingTrust Wang et al. (2024) is a framework for evaluating the trustworthiness of LLMs across eight dimensions, one of which is stereotype bias. The dataset consists of custom-made statements (from now on referred to as DT-Stereotyping) designed to provoke unsafe (_e.g._, biased, toxic) responses. Following Wang et al. (2024), we ask the model to either agree or disagree with these statements. To measure stereotyping behavior, we compute the fraction of times the model agrees with the stereotypical statements \(\frac{n_{\text{stereo}}}{n}\in[0,1]\). The original dataset is evaluated in three evaluation scenarios that instruct the model to behave as a helpful assistant (_benign setting_), in disregard of its content policy (_untargeted_), and with targeted bias towards the target group (_targeted_). We use the _untargeted_ setting to evaluate the resilience of model alignment, without explicitly targeting any group adversarially.

Since model outputs can be generated with different temperatures (\(T\)), we use two variants of this evaluation. We refer to these as DT-Stereotyping (greedy) with \(T=0\) and DT-Stereotyping (sampling) with \(T=1\) and top-p \(=1\).

**DiscrimEval**Tamkin et al. (2023) consists of 70 hypothetical decision making scenarios, _e.g._, approving a loan. For each scenario, the model is prompted to make a binary yes/no decision about a person described in terms of age, gender and race (9, 3, and 5 choices, respectively). A "yes" decision is always advantageous. Following Tamkin et al. (2023), we append "My answer would be" to the prompt to steer the generations towards producing binary decisions and record the model's softmax probability of "yes" or "no" being generated as the first token. For a scenario \(q_{i}\), and a set of demographic groups \(G\) (\(9\times 3\times 5=135\) intersectional groups in this case), we repeatedly reformulate \(q_{i}\), substituting the demographic information for all groups \(g\in G\) one by one, and measure the difference between the highest and lowest probability of "yes" for all groups \(g\in G\). Specifically, the bias score is:

\[\tfrac{1}{n}\sum_{q_{i}\in Q}\big{(}\max_{g\in G}P(\text{yes}|q_{i},g)-\min_{ g\in G}P(\text{yes}|q_{i},g)\big{)}\in[0,1],\]

where \(Q\) is the set of all questions and \(n=|Q|\). We use the "explicit" version of the dataset in our evaluation, directly including demographic attributes in the prompt rather than implying it via names.

**DiscrimEvalGen.** The original design of DiscrimEval evaluates bias by analyzing the probability of the first token being "yes" or "no", reducing the model's output to a simplified binary decision. This **approach has the following limitations**:

1. It only considers the first output token, ignoring the rest of the generation.
2. Even at the first token, the bias is computed by considering the Softmax probabilities of 'yes' and 'no' which could be miscalibrated and may not adequately capture the model's uncertainty Cruz et al. (2024).
3. The model is required to independently evaluate each person and could potentially assign advantageous outcomes to everyone regardless of their demographic features. Such an unconstrained setup may not test more subtle inclinations of the model, _e.g._, preferences when required to make a choice between different demographic groups or refusing to make a choice at all. In fact, Bai et al. (2024) show that GPT-4 assigns benefits to various demographic groups in similar proportions.

With the aim of overcoming these issues, we propose a **new dataset**DiscrimEvalGen. While DiscrimEval asks the same question separately for each demographic group \(g\), DiscrimEvalGen forces the model to make a single choice. Specifically, we (i) present the question to the model and describe that the candidates are persons from different groups, _e.g._, a man, a woman, a non-binary person; (ii) describe that the benefit (_e.g._, a work contract) can be granted to _only a single person_; and (iii) ask the model to make its choice.

Let \(q\in Q\) be the questions, \(g\in G\) be the groups, and \(n_{g}\) be the number of times a group is selected by the model with \(\sum_{g\in G}n_{g}=|Q|\), then the bias metric is:

\[\frac{1}{n}\big{(}\max_{g\in G}n_{g}-\min_{g\in G}n_{g}\big{)}\in[0,1].\]

Figure A.1 in Appendix A shows a concrete example. To avoid having a very long list of choices (\(135\) intersectional groups in the original dataset), we limit the groups to those based on gender, that is, \(G=\{\text{man},\text{non-binary},\text{woman}\}\). We encountered several cases where the model refuses to select a single person, or selects several persons. We ignore such cases from the bias computation. If for a



particular model/acceleration strategy combination, we have more than \(80\%\) such cases, we drop this combination from our results.

Just like DT-Stereotyping, we consider two versions: DiscrimEvalGen (greedy) with \(T=0\) and DiscrimEvalGen (sampling) with \(T=1\) and top-p \(=1\).

## 4 Experimental Setup

**Models and Infrastructure.** We analyze three different models: LLaMA-2 Touvron et al. (2023), LLaMA-3.1 Dubey et al. (2024), and Mistral-0.3 Jiang et al. (2023). We consider the smallest size variant of each model: LLaMA-2-7B, LLaMA-3.1-8B, and Mistral-7B-v0.3 license information in Appendix A). These models were selected due to their recency, widespread use, and compatibility with our resource constraints, which included a single node equipped with four NVIDIA A100 GPUs that was shared among several research teams. Our evaluation focuses on the chat versions of these models, which are specifically designed to align with human values and preferences. We used the GitHub Copilot IDE plugin to assist with coding.

**Inference acceleration strategies.** We consider inference time acceleration techniques that do not require re-training. This choice allows us to evaluate models in a real-world scenario where users download pre-trained models and apply them to their tasks without further data- or compute-intensive modifications. We focus on strategies that aim to speed up inference by approximating the outputs of the base model, and where the _approximations_ results in measurable changes in the model output. This criterion excludes strategies like speculative decoding Leviathan et al. (2023) where the output of the base and inference accelerated models are often the _same_. Specifically, we consider the following strategies:

**Quantization.** We consider the following variants:

1. [noitemsep]
2. **INT4** or **INT8** quantization using Bitsandbytes library Bit (2024) which first normalizes the model weights to store common values efficiently. Then, it quantizes the weights to 4 or 8 bits for storage. Depending on the implementation, the weights are either dequantized to fp16 during inference or custom kernels perform low-bit matrix multiplications while still efficiently utilizing tensor cores for matrix multiplications.
3. Activation-aware Weight Quantization (**AWQ**) Lin et al. (2024) quantizes the parameters by taking into account the data distribution in the activations produced by the model during inference. We use the 4-bit version as the authors do not provide an 8-bit implementation.
4. Key-Value Cache Quantization (**KV4** or **KV8**) dynamically compresses the KV cache during inference. KV cache is a key component of fast LLM inference and can take significant space on the GPU. Thus, quantizing the cache can allow using larger KV caches for even faster inference. We use both 4 and 8-bit quantization Liu et al. (2023). We use the native HuggingFace implementation. This implementation does not support Mistral models.

**Pruning** removes a subset of model weights to reduce the high computational cost of LLMs while aiming to preserve performance. Traditional pruning methods require retraining Cheng et al. (2024). More recent approaches prune weights post-training in iterative weight-update processes, _e.g._, SparseGPT Frantar and Alistarh (2023). We use the Wanda method by Sun et al. (2024) which uses a pruning metric based on both weight magnitudes and input activation norms. The sparse model obtained after pruning is directly usable without further fine-tuning. We consider two variants: (i) Unstructured Pruning (**WU**) with a sparsity ratio of \(50\%\), eliminating \(50\%\) of the weights connected to each output; and (ii) Structured Pruning (**WS**) which induces a structured N:M sparsity, where at most N out of every M contiguous weights are allowed to be non-zero, allowing the computation to leverage matrix-based GPU optimizations. We use a \(2:4\) compression rate. Prior work has shown that pruned models can preserve comparable performance levels even at high compression rates Frantar and Alistarh (2023); Jaiswal et al. (2024); Sun et al. (2024), _e.g._, \(2:4\) considered here.

**Parameters.** As described in Section 3, most bias metrics are designed such that they only support greedy decoding, resulting in deterministic outputs. Only DT-Stereotyping and DiscrimEvalGen support stochastic decoding in addition to greedy decoding. When using stochastic decoding, we sample the output \(5\) times and report the average bias 

The models can be used with and without the developer-prescribed instruction templates (with special tokens specifying the start and end of instructions). Past investigations have shown that instruction and answer formats can have an unpredictable impact on the model performance Alzahrani et al. (2024); Fourrier et al. (2023). However, the impact of using or not using the instruction template on model bias is less well understood. We thus study both configurations. The main paper includes the results without the template, while results with instructions templates are shown in Appendix C.

## 5 Results

Table 2 shows the bias of base models w.r.t. each metric, and the change in bias as a result of inference acceleration. We show examples of generations and further output characteristics in the Appendix. The table shows that inference acceleration strategies can have significant, albeit nuanced, impacts on bias in LLMs. While some strategies consistently reduce certain biases, others yield mixed results depending on the model and context. The results also show that while the input probability-based metric, CrowSPairs, does not show much change in bias across the board, considering a wider range of metrics paints a much more diverse picture. While the exact magnitude of changes varies, we largely see similar trends of unpredictable effects on downstream bias both with and without the instruction template (Appendix C). Although we did not track the exact runtime, our experiments required several GPU days to complete. We now analyze each RQ from Section 1 in detail.

**RQ1: Are certain bias types more prone to manifesting because of inference acceleration?**

Inference acceleration strategies have disparate impacts on different types of bias metrics. Specifically, we see:

_No significant impact on log-likelihood of stereotypical sentences_ as measured by the CrowsPairs dataset. Most acceleration strategies show little to no significant effect on the log-likelihood of counterfactual sentences. The results are largely in line with Goncalves and Strubell (2023) who also show a relatively mild effect of quantization on bias measured via CrowSPairs, although they consider the previous generation of LMs like BERT and RoBERTa. We provide a detailed breakdown of results per bias type in Table B.1. Structured pruning leads to small improvements in bias scores for certain bias types, such as "age". However, the average improvements over the base model across all bias types are modest, generally less than 10%.

_Subtle shifts in values and opinions in the GlobalOpinionQA task._ We observe little effect of inference strategies on the values and opinions represented by the models (see Table 2b). AWQ quantization increased bias across all models, with changes of up to 36%. Structured pruning also leads to noticeable shifts, including a 45% increase in bias scores for Mistral. Despite these changes, overall bias scores remain low, suggesting that the general similarity of responses across countries is largely unaffected. Notably, KV cache quantization shows no negative impact. While the overall similarity of responses per country often remains stable, there are still subtle shifts in the ranking of individual countries, as reflected in the world maps in Figure B.1.

_Pruning influences models' ability to recall country-specific facts._ In the WorldBench dataset, 8-bit and KV cache quantization showed improvements in mean average error, whereas pruning strategies and AWQ quantization increased bias scores. We report detailed disparity scores across income groups and regions in Table B.5. Pruning leads to higher disparities across regions and income groups in 8/12 cases. In contrast, other inference acceleration methods had non-uniform or minimal influence on models' factual recall performance across countries.

_More pronounced shifts in model's agreement with stereotypes._ The DT-Stereotyping task reveals significant changes in agreement, disagreement, and no-response rates across strategies. Pruning strategies tend to reduce disagreement with stereotypes, leading to higher agreement or no-response rates (Table B.2). Quantization showed minimal effects or slight improvements for LLMA models but increased the number of agreements with stereotypes for Mistral. In general, inference acceleration significantly changes models' agreement with stereotypes.

_Varying bias patterns in allocation-based decision-making scenarios._ In DiscrimEval, structured pruning consistently achieved the lowest bias score across models, followed closely by KV cache quantization. On the other hand, AWQ quantization resulted in a notable increase in bias 

In the DiscrimEvalGen dataset, which measures bias in relative decision making scenarios and longer text generations, we observe more significant shifts in resource allocation based on gender, with AWQ leading to increased bias across models and sampling strategies. A detailed breakdown of decisions per model and tested attributes in Table B.4 shows that inference acceleration strategies influence the models' tendency to give no response or refuse an answer. Both Mistral and LLaMA-3.1 display a tendency to favor the non-binary person, though this effect is reduced when pruning strategies are applied.

**RQ2: Are certain inference acceleration strategies more prone to bias?**

Table 2 shows that the change in bias heavily depends on the acceleration strategy. Notably, AWQ quantization performed worse than suggested by recent work Hong et al. (2024), leading to massively increased bias in DiscrimEval

\begin{table}
\begin{tabular}{l c r r r r r r r r r r} \hline \hline  & Base & WS & WU & AWQ & INT4 & INT8 & KV4 & KV8 \\ \hline LLaMA-2 & 0.52 & \(\dagger\)98 0.56 & \(\dagger\)76 & \(\dagger\)10 & \(\dagger\)10 & \(\dagger\)10 & \(\dagger\)10 & \(\dagger\)10 & \(\dagger\)10 & \(\dagger\)10 & \(\dagger\)10 \\ Mistral & 0.43 & \(\dagger\)32 0.59 & \(\dagger\)19 0.51 & \(\dagger\)20 & \(\dagger\)44 & \(\dagger\)0.43 & \(\dagger\)21 0.34 & NI & NI & NI \\ LLaMA-3.1 & 0.4 & \(\dagger\)25 0.5 & \(\dagger\)38 0.55 & \(\dagger\)10 0.44 & \(\dagger\)55 0.42 & \(\dagger\)33 0.39 & 0.4 & NI & 0.39 \\ \hline \multicolumn{11}{c}{(c) WorldBench} \\ \hline  & Base & WS & WU & AWQ & INT4 & INT8 & KV4 & KV8 \\ \hline LLaMA-2 & 0.22 & \(\dagger\)38 0.03 & \(\dagger\)27 0.16 & \(\dagger\)123 0.49 & \(\dagger\)38 0.14 & \(\dagger\)18 0.26 & \(\dagger\)64 0.08 & 0.22 \\ Mistral & 0.1 & \(\dagger\)40 0.06 & \(\dagger\)10 0.09 & \(\dagger\)110 0.21 & \(\dagger\)10 0.09 & \(\dagger\)110 0.11 & NI & NI & NI \\ LLaMA-3.1 & 0.19 & \(\dagger\)38 0.08 & \(\ddagger\)47 0.1 & \(\dagger\)110 0.21 & \(\dagger\)55 0.2 & \(\dagger\)26 0.24 & \(\dagger\)38 0.08 & 0.19 \\ \hline \end{tabular}
\end{table}
Table 2: Effect of inference acceleration on bias. Each subtable shows a different bias metric from Section 3. The first column shows the bias of the base model without any acceleration. Each cell displays the absolute bias value along with the percentage change relative to the base model. A value of \(\#\)x or \(\#\)x represents a \(X\%\) increase or \(Y\%\) decrease in bias w.r.t. the base model. A value of NI means the acceleration strategy is not implemented for that model. A value of ND means there was not enough data for this combination (see Section 3). Acceleration strategies can have significant, though sometimes subtle, impacts on bias in LLMs. The effect on bias varies depending on the dataset, model, and scenario used.

 

scenarios for LLaMA-2 and Mistral, and heightened agreement with stereotypical statements in DT-Stereotyping for Mistral. While previous work by Hong et al. suggested that quantization is an effective compression technique with minimal impact on trustworthiness, our findings highlight the need to evaluate these strategies across multiple models and evaluation contexts to capture their broader effects.

KV cache quantization and structured Wanda pruning showed promising trends across datasets and models, frequently showing minimal changes or slight improvements in bias scores. However, structured pruning exhibited certain drawbacks. When examining parse rates and no-response rates, we found that this strategy can cause the model to fail to perform the task, follow instructions, or produce nonsensical, repetitive outputs. _Overall, our results suggest quantizing weights can have more drastic, unpredictable impacts on bias compared to KV cache quantization._

**RQ3: Does the bias impact of a strategy remain consistent across models?**

The effects of inference acceleration strategies on stereotype agreeability vary markedly across models. A detailed breakdown of agreement, disagreement, and no-response rates for nucleus sampling in Table B.2 illustrates how the models' baselines already differ. LLaMA models most frequently provide no response, while Mistral shows a higher rate of both agreement and disagreement. Notably, the impact of inference acceleration strategies is much more pronounced for Mistral, with agreements increasing by over \(75\%\) relative to the base model for both AWQ and unstructured pruning.

Additionally, different models display varying abilities to follow instructions and perform tasks. For example, in the DiscrimEvalGen dataset [10], LLaMA-2 mostly provides no response. Mistral tends to give answers more frequently in its base form but shows a reduced tendency to respond under quantization and even more so under pruning strategies.

Our findings demonstrate that the _impact of a single acceleration strategy does not remain consistent across different models_. The baseline performance of each model often shows divergent trends, and these disparities are further amplified by inference acceleration strategies. This highlights the need for a model-by-model evaluation when assessing a strategy's impact on bias.

Comparing 4-bit and 8-bit compressionWhile lower-bit compression can enhance efficiency, it often risks degrading model performance [6]. Hong et al. (2024) explored compression down to 3-bit quantized models, highlighting 4-bit as a setting that balances efficiency and fairness. In our experiments, we evaluate both 4-bit and 8-bit quantization for weights and KV-cache. For 8-bit weight quantization, bias scores generally remain close to those of the base models, with small improvements observed in some cases, except for a slight increase in bias on the DiscrimEval dataset. Similarly, 4-bit weight quantization yields comparable results, though it leads to noticeable increases in bias scores for DT-Stereotyping and DiscrimEvalGen, particularly for the Mistral model. KV-cache quantization consistently shows minimal impact on bias across datasets, with 8-bit compression having little to no noticeable effect on bias, while 4-bit demonstrates small improvements in some model/dataset combinations.

Combining inference acceleration strategiesWe also explore the impact of multiple inference acceleration strategies on model bias. In Table 3, we compare INT4 quantization, 4-bit KV cache quantization, and a combination of both. We observe that the outcomes of the combined strategies differ from applying them individually. In some cases, the bias observed aligns with INT4 quantization (_e.g._, for DiscrimEvalEval), in others with KV cache quantization (_e.g._, for DiscrimEvalGen). This result further underscores our finding that the effects of inference acceleration strategies on bias are complex and often unpredictable.

Effects of using provider-prescribed instruction templatesWe study whether using provider-prescribed instruction templates ameliorates the bias resulting from inference acceleration. We re

\begin{table}
\begin{tabular}{l|r r r r} Dataset & Base & INT4 & KV4 & Comb \\ \hline GlobalOpinionQA & 0.14 & 0.14 & 0.14 & 0.14 \\ WorldBench & 0.4 & \(\lessapprox\) 0.42 & 0.4 & \(\lessapprox\) 0.42 \\ DiscrimEval & 0.19 & \(\lessapprox\) 0.23 & \(\lessapprox\) 0.08 & \(\lessapprox\) 0.2 \\ DT-Stereo **(g)** & 10 & \(\lessapprox\) 10 & 10 & \(\lessapprox\) 2 \\ DT-Stereo **(s)** & 20 & \(\lessapprox\) 9 & \(\lessapprox\) 21 & \(\lessapprox\) 11 \\ DiscrimEvalGen **(g)** & 0.61 & \(\lessapprox\) 0.74 & \(\lessapprox\) 0.51 & \(\lessapprox\) 0.53 \\ DiscrimEvalGen **(s)** & 0.16 & \(\lessapprox\) 0.23 & \(\lessapprox\) 0.09 & \(\lessapprox\) 0.19 \\ \end{tabular}
\end{table}
Table 3: Comparison of INT4, KV4, and the combination of both (Comb) on LLaMA-3.1 **(g:** greedy, **s:** sampling). Combinations of inference acceleration strategies also lead to unpredictable effects on bias.

 

port the results with the developer-prescribed instruction templates in Appendix C. We do not include the CrowSPairs data since the addition of instruction tokens means that we can no longer measure the exact log-likelihood of the input sentences. The results show largely similar trends as in Table 2. However, in some cases (_e.g._, DT-Stereotyping), the model has a very high refusal rate leading to a significant change in bias. These findings further emphasize the need for a careful bias analysis before deploying accelerated models.

**Effects of inference acceleration on text characteristics beyond bias.** In addition to bias, we observe that inference acceleration can alter fundamental text characteristics, such as response length. Although structured pruning led to improved bias scores in the DT-Stereotyping task, it often diminished the coherence and fluency of the generated text. Examples of this behavior are shown in Table B.3. A detailed analysis of text characteristics, provided in Appendix B, shows that deployment strategies can significantly affect aspects of text generation beyond bias. For instance, structured pruning increases the average response length in LLaMA-2 from 65 to 107 words. For LLaMa-3.1, the rate of non-dictionary words increases from 11% to 25%. These varied effects highlight the need to evaluate these strategies holistically rather than solely relying on standard benchmarks.

## 6 Conclusion & Future Work

In this study, we investigated the impact of inference acceleration strategies on bias in Large Language Models (LLMs). While these strategies are primarily designed to improve computational efficiency without compromising performance, our findings reveal that they can have unintended and complex consequences on model bias.

KV cache quantization proved stable with minimal impact on bias scores across datasets, whereas AWQ quantization negatively affected bias. Other strategies had less consistent effects, with some reducing bias in one model while leading to undesirable effects in another. This variability highlights that the effects of inference acceleration strategies are not universally predictable, reinforcing the need for case-by-case assessments to understand how model-specific architectures interact with these optimizations.

The impact of these strategies extends beyond bias. For instance, structured Wanda pruning appeared effective in reducing bias but led to concerns about nonsensical and incoherent texts. Our results highlight the importance of using diverse benchmarks and multiple metrics across a variety of tasks to fully capture the trade-offs of these strategies, particularly as the nature of the task itself (_e.g._, generation vs probability-based) can surface different kinds of biases.

Bias mitigation is an important direction for future research. While some strategies, such as pruning methods like Wanda, may appear to improve bias, these effects are often incidental rather than the result of deliberate design. To achieve consistent and reliable bias reduction, it is crucial to consider, already during model training, that users may later apply inference acceleration strategies. Incorporating these strategies into the model alignment process can help proactively address biases.

It may also be useful to explore approaches that integrate explicit bias mitigation objectives, such as fairness-aware training methods or bias-sensitive hyperparameter optimization Agarwal et al. (2018); Perrone et al. (2021); Raj et al. (2024). Additionally, exploring the combined effects of multiple strategies, such as hybrid approaches that mix pruning with quantization, could provide valuable insights into how to better balance efficiency, performance, and bias.

Our analysis focused on demographic bias. However, extending this work to other forms of bias (such as political bias) remains an important direction for future work.

## 7 Limitations

Our study has several limitations that should be taken into account when interpreting the results. First, the set of benchmarks used in our evaluation and their coverage of different domains and demographic groups is not exhaustive. Since our metrics do not cover all manifestations of bias, there is a risk that some inference acceleration strategies may appear to be less prone to bias based on the chosen metrics, while in reality, they may exhibit nuanced, domain-specific biases not measured here. Specifically, demographic bias in LLMs encompasses a wide range of groups (_e.g._, based on age, gender, race), manifests in various ways, and can cause different types of harm. Addressing these biases requires diverse measurement approaches Gallegois et al. (2024); Mehrabi et al. (2022) 

Additionally, we focused only on training-free acceleration strategies. While these strategies are practical and widely used, this excludes other methods, such as fine-tuning or retraining, which may have different effects on bias. Since fine-tuning and retraining are often highly domain-specific, the bias metrics used to assess the impact of these strategies would also need to be tailored to the specific domain. Furthermore, our use of fixed hyperparameters (_e.g._, greedy search, sampling five generations) may not capture the full range of model behaviors under different deployment conditions.

There are also potential risks associated with our findings. One risk is that users might interpret our results as suggesting that some deployment strategies are inherently free of bias, which is not the case. Given the limitations of our study, our results should be taken as indicative rather than definitive since bias in modern, instruction-tuned LLMs remains an under-explored area [24].

Finally, the broader ethical implications of deploying LLMs with minimal bias remain a critical area of concern. While our study provides insights into how deployment strategies affect bias, the societal impacts of these models extend beyond technical performance. Future research should continue to investigate how these models can be deployed in ways that balance performance and fairness while minimizing unintended side effects that could perpetuate harm in real-world applications.

## References

* Bitsandbytes (2024) Bitsandbytes. Note: [https://huggingface.co/docs/transformers/main/en/quantization/bitsandbytes](https://huggingface.co/docs/transformers/main/en/quantization/bitsandbytes) Cited by: SS1.
* The AI community building the future ([https://huggingface.co/](https://huggingface.co/). Cited by: SS3.
* A. Agarwal, A. Beygelzimer, M. Dudik, J. Langford, and H. Wallach (2018) A reductions approach to fair classification. In Proceedings of the 35th International Conference on Machine Learning, Vol. 80, pp. 60-69. External Links: Link Cited by: SS1.
* N. Alzhrani, H. Alyahya, Y. Alnumay, S. AlRashed, S. Alsubaie, Y. Almushayqih, F. Mirza, N. Alotaibi, N. AlTwairesh, A. Alowisheq, M. Bari, and H. Khan (2024) When benchmarks are targets: Revealing the sensitivity of large language model leaderboards. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangok, Thailand, pp. 13787-13805. External Links: Link Cited by: SS1.
* M. Arjovsky, S. Chintala, and L. Bottou (2017) Wasserstein GAN. Preprint arXiv:1701.07875. External Links: Link Cited by: SS1.
* X. Bai, A. Wang, I. Sucholutsky, and T. L. Griffiths (2024) Measuring Implicit Bias in Explicitly Unbiased Large Language Models. PreprintarXiv:2402.04105. External Links: Link Cited by: SS1.
* S. Lin Blodgett, S. Barocas, H. Daume III, and H. Wallach (2020)Language (technology) is Power: A Critical Survey of "Bias" in NLP. PreprintarXiv:2005.14050. External Links: Link Cited by: SS1.
* S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, H. Nori, H. Palangi, M. Tulio Ribeiro, and Y. Zhang (2023)Sparks of Artificial General Intelligence: Early experiments with GPT-4. Note: [https://arxiv.org/abs/2303.12712v5](https://arxiv.org/abs/2303.12712v5) External Links: Link Cited by: SS1.
* A. Caliskan, J. J. Bryson, and A. Narayanan (2017)Semantics derived automatically from language corpora contain human-like biases. Science356 (6334), pp. 183-186. External Links: ISSN 0037-69X, Document, Link Cited by: SS1.
* H. Cheng, M. Zhang, and J. Qinfeng Shi (2024)A survey on deep neural network pruning- Taxonomy, comparison, analysis, and recommendations. PreprintarXiv:2308.06767. External Links: Link Cited by: SS1.
* A. F. Cruz, M. Hardt, and C. Mendler-Dunner (2024)Evaluating language models as risk scores. PreprintarXiv:2407.14614. External Links: Link Cited by: SS1.
* T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer (2022)LLM.in\(8\)(Website) 8-bit Matrix Multiplication for Transformers at Scale. PreprintarXiv:2208.07339. External Links: Link Cited by: SS1.
* J. Dhamala, T. Sun, V. Kumar, S. Krishna, Y. Prukaschatkun, K. Chang, and R. Gupta (2021) BOLD: Dataset Metrics for Measuring Biases in Open-Ended Language Generation. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 862-872. External Links: ISBN 978145036687, Link, Document Cited by: SS1.
* A. Dubey et al. (2024)The llama 3 herd of models. PreprintarXiv:2407.21783. External Links: Link Cited by: SS1.
* E. Durmus, K. Nguyen, T. I. Liao, N. Schiefer, A. Askell, A. Bakhin, C. Chen, Z. Hatfield-Dodds, D. Hernandez, N. Joseph, L. Lovitt, S. McCandlish, O. Sikder, A. Tamkin, J. Thamkul, J. Kaplan, J. Clark, and D. Ganguli (2024)Towards measuring the representation of Subjective Global Opinions in Language Models. PreprintarXiv:2306.16388.

 

Clementine Fourrier, Nathan Habib, Julien Launay, and Thomas Wolf. 2023. Bitsandbytes. [https://huggingface.co/blog/open-llm-leaderboard-mmlu](https://huggingface.co/blog/open-llm-leaderboard-mmlu).
* Frantr and Alistarh (2023) Elias Frantr and Dan Alistarh. 2023. SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. _Preprint_, arXiv:2301.00774.
* Gallegos et al. (2024) Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. 2024. Bias and Fairness in Large Language Models: A Survey. _Preprint_, arXiv:2309.00770.
* Goncalves and Strubell (2023) Gustavo Goncalves and Emma Strubell. 2023. Understanding the Effect of Model Compression on Social Bias in Large Language Models. _Preprint_, arXiv:2312.05662.
* Hartvigsen et al. (2022) Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection. _Preprint_, arXiv:2203.09509.
* Hong et al. (2024) Junyuan Hong, Jinhao Duan, Chenhui Zhang, Zhangheng Li, Chulin Xie, Kelsey Lieberman, James Diffenderfer, Brian Bartoldson, Ajay Jaiswal, Kaidi Xu, Bhavya Kailkura, Dan Hendrycks, Dawn Song, Zhangyang Wang, and Bo Li. 2024. Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression. _Preprint_, arXiv:2403.15447.
* Hooper et al. (2024) Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. 2024. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization. _Preprint_, arXiv:2401.18079.
* Huang et al. (2024) Hai Huang, Zhengyu Zhao, Michael Backes, Yun Shen, and Yang Zhang. 2024. Composite Backdoor Attacks Against Large Language Models. _Preprint_, arXiv:2310.07676.
* Jaiswal et al. (2024) Ajay Jaiswal, Zhe Gan, Xianzhi Du, Bowen Zhang, Zhangyang Wang, and Yinfei Yang. 2024. Compressing LLMs: The Truth is Rarely Pure and Never Simple. _Preprint_, arXiv:2310.01382.
* Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lampe, Lucile Sauhler, Lelio Renard Lavaud, Marie-Anne Lachatx, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothee Lacroix, and William El Sayed. 2023. Mistral 7B. _Preprint_, arXiv:2310.06825.
* Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with megategmentation. In _Proceedings of the 29th Symposium on Operating Systems Principles_, SOSP '23, page 611-626, New York, NY, USA. Association for Computing Machinery.
* Leviathan et al. (2023) Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast Inference from Transformers via Speculative Decoding. _Preprint_, arXiv:2211.17192.
* Lin et al. (2024) Ji Lin, Jianqing Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dong, Chuang Gan, and Song Han. 2024. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. _Preprint_, arXiv:2306.00978.
* Liu et al. (2023) Zirui Liu, Jiaqi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2023. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache.
* May et al. (2019) Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On Measuring Social Biases in Sentence Encoders. _Preprint_, arXiv:1903.10561.
* Mehrabi et al. (2022) Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2022. A Survey on Bias and Fairness in Machine Learning. _Preprint_, arXiv:1908.09635.
* Mo et al. (2024) Lingbo Mo, Boshi Wang, Muhao Chen, and Huan Sun. 2024. How Trustworthy are Open-Source LLMs? An Assessment under Malicious Demonstrations Shows their Vulnerabilities. _Preprint_, arXiv:2311.09447.
* Moayeri et al. (2024) Mazda Moayeri, Elham Tabassi, and Soheil Feizi. 2024. WorldBench: Quantifying Geographic Disparities in LLVM Factual Recall. In _The 2024 ACM Conference on Fairness, Accountability, and Transparency_, pages 1211-1228, Rio de Janeiro Brazil. ACM.
* Nangia et al. (2020) Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-Paris: A Challenge Dataset for Measuring Social Biases in Masked Language Models. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1953-1967, Online. Association for Computational Linguistics.
* Park et al. (2024) Seungcheol Park, Jaehyeon Choi, Sojin Lee, and U. Kang. 2024. A Comprehensive Survey of Compression Algorithms for Language Models. _Preprint_, arXiv:2401.15347.
* Perez et al. (2022) Ethan Perez et al. 2022. Discovering Language Model Behaviors with Model-Written Evaluations. _Preprint_, arXiv:2212.09251.
* Perrone et al. (2021) Valerio Perrone, Michele Donini, Muhammad Bilal Zarfar, Robin Schmuker, Krishnaram Kenthapadi, and Cedric Archambbeau. 2021. Fair bayesian optimization. In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_, pages 854-863 

Cited by: SS2.
* C. Raj, A. Mukherjee, A. Caliskan, A. Anastasopoulos, and Z. Zhu (2024)Breaking bias, building bridges: evaluation and mitigation of social biases in llms via contact hypothesis. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, Vol. 7, pp. 1180-1189. Cited by: SS3.1.
* E. Michael Smith, M. Hall, M. Kambadur, E. Presani, and A. Williams (2022)"I'm sorry to hear that": finding new biases in language models with a holistic descriptor dataset. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, pp. 9180-9211. External Links: Link, Document Cited by: SS5.
* G. Raj, A. Mukherjee, A. Caliskan, A. Anastasopoulos, and Z. Zhu (2024)Breaking bias, building bridges: evaluation and mitigation of social biases in llms via contact hypothesis. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, Vol. 7, pp. 1180-1189. Cited by: SS3.1.
* H. Suresh and J. V. Guttag (2021)A framework for understanding sources of harm throughout the machine learning life cycle. In Equity and Access in Algorithms, Mechanisms, and Optimization, pp. 1-9. Cited by: SS1.
* A. Tamkin, A. Askell, L. Lovitt, E. Durmus, N. Joseph, S. Kravec, K. Nguyen, J. Kaplan, and D. Ganguli (2023)Evaluating and mitigating discrimination in language model decisions. PreprintarXiv:2312.03689. Cited by: SS1.
* H. Touvron et al. (2023)Llama 2: open foundation and fine-tuned chat models. PreprintarXiv:2307.09288. Cited by: SS1.
* P. et al. Virtanen et al. (2020)SciPy 1.0: fundamental algorithms for scientific computing in python. Nature Methods17, pp. 261-272. Cited by: SS1.
* B. Wang, W. Chen, H. Pei, C. Xie, M. Kang, C. Zhang, C. Xu, Z. Xiong, R. Dutta, R. Schaeffer, S. T. Truong, S. Arora, M. Mazeika, D. Hendrycks, Z. Lin, Y. Cheng, S. Koyejo, D. Song, and B. Li (2024)DecodingTrust: a comprehensive assessment of trustworthiness in GPT models. PreprintarXiv:2306.11698. Cited by: SS1.
* J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, Ed H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus (2022)Emergent abilities of large language models. PreprintarXiv:2206.07682. Cited by: SS1.
* G. Xu and Q. Hu (2022)Can model compression improve NLP fairness. PreprintarXiv:2201.08542. Cited by: SS1.
* X. Zhu, J. Li, Y. Liu, C. Ma, and W. Wang (2023)A survey on model compression for large language models. Note: [https://arxiv.org/abs/2308.07633v4](https://arxiv.org/abs/2308.07633v4) Cited by: SS1.
* C. Ziems, W. Held, O. Shaikh, J. Chen, Z. Zhang, and D. Yang (2024)Can large language models transform computational social science?. PreprintarXiv:2305.03514 Cited by: SS1.

 



## Appendix A Additional Reproducibility Details

Table A.1 provides additional details like number of prompts and the types of bias being measured for each dataset.

**Dataset Licenses and Usage.** All datasets were released with the goal of measuring bias so our usage complies with their intended use.

1. **CrowSPairs:** We use the dataset version provided by the authors at [https://github.com/nyu-mll/crows-pairs](https://github.com/nyu-mll/crows-pairs). The authors provided the dataset under a CC BY-SA 4.0 license.
2. **DiscrimEval:** We use the dataset version provided by the authors at [https://huggingface.co/datasets/Anthropic/discrim-eval](https://huggingface.co/datasets/Anthropic/discrim-eval). The authors provided the dataset under a CC-BY-4.0 license.
3. **DiscrimEvalGen:** We derived this dataset from DiscrimEval (Section 3). We will make the dataset publicly available under the same license.
4. **GlobalOpinion:** We use the dataset version provided by the authors at [https://huggingface.co/datasets/Anthropi](https://huggingface.co/datasets/Anthropi) c/llm_global_opinions. The authors provided the dataset under a CC-BY-NC-SA-4.0 license.
5. **DT-Stereotyping:** We use the dataset version provided by the authors at [https://github.com/AI-secure/DecodingTrus](https://github.com/AI-secure/DecodingTrus) t. The authors provided the dataset under a CC-BY-SA-4.0 license.
6. **WorldBench:** We use the dataset version provided by the authors at [https://github.com/mnoayeri/world-bench/tree](https://github.com/mnoayeri/world-bench/tree) e/main. The authors did not provide a license. However, the dataset was copied from the WorldBank website who make it available under a CC-BY 4.0 license ([https://datacatalog.wordbank.org/public-licenses](https://datacatalog.wordbank.org/public-licenses)).

**Model Licenses.** We use the model implementations from original providers at the HuggingFace Hub, namely, mistralai/Mistral-7B-Instruct-v0.3, meta-llama/Llama-2-?b-chat-hf and meta-llama/Llama-3.1-8B-Instruct. Mistral provides the model under an Apache-2.0 licenses. Meta provides the models under custom llama2 and llama3.1 licenses.

## Appendix B Additional Results

### Analysis of Text Characteristics

To get a better understanding of the models' generations and how they are affected by inference acceleration strategies, we analyze generations on the DT-stereotyping benchmark averaged on 5 generations with nucleus sampling. We compute the following metrics:

1. **Average Response Length (ARL):** We compute the average response length as the mean number of words in the generated text to assess the models' verbosity, using the word-tokenize function from the Natural Language Toolkit (NLTK) library.1 Footnote 1: [https://www.nltk.org/](https://www.nltk.org/)
2. **Average Non-Dictionary Word Rate (ANDWR):** This metric calculates the average proportion of non-dictionary words in the generated texts. As a reference dictionary, we use the _words_ corpus from NLTK.
3. **Average Repetition Rate (ARR):** We measure the average number of repeated words in the generated text to analyze repetitiveness and redundancy in the generated texts.
4. **Average Lexical Diversity (ALD):** Lexical diversity is a measure of the richness of the vocabulary used in a text. The metric is computed as the ratio of the number of unique words to the total number of words in the generated text.

We report these metrics in Table A.2 We observe that the baselines of the different models show different response lengths, with LLaMA-3.1 generating texts twice as long as LLaMA-2. The response length for LLaMA-2 increases significantly when pruning strategies are applied. For Mistral, we observe a decrease in response length when applying unstructured pruning or quantization. Regarding non-dictionary words, ANDWR is relatively low

across all models and deployment strategies, indicating that the generated texts are mostly composed of existing English words. ANDWR is highest for LLaMA-3.1 when applying structured wanda pruning with 25% of the words not found in the dictionary. We give examples of the generated texts for LLaMA-3.1 in Table B.3. We see that the model is able to generate full sentences in greedy search, but the text quality deteriorates significantly when using nucleus sampling. The generated texts are incoherent and contain non-dictionary words, indicating that the effect of structured pruning on the coherence of the generated texts is impacted by the sampling method. For LLaMA-3.1, we observe a higher repetition rate and a lower lexical diversity than for the other models. KV-Cache quantization shows no significant impact on the characteristics of the generated texts with results similar to the baselines.

To summarize, we observe that deployment strategies can have a significant impact on the fundamental characteristics of the generated texts, such as repetitive content, non-dictionary words, and lexical diversity. These effects vary remarkably across models and deployment strategies, indicating that the impact of deployment strategies on the text characteristics is model-dependent and non-trivial. While quantization shows little impact on the generated texts, pruning can significantly impact the coherence and meaningfulness of model generations.

## Appendix C Results With Instruction / Chat Template

It is essential to evaluate LLMs not only within prescribed frameworks but also across a range of possible usage scenarios to fully understand their behavior in diverse contexts. While the use of chat templates is often advised, it is unclear whether businesses and end users consistently adopt this

\begin{table}
\begin{tabular}{l c l} \hline \hline Dataset & \#Prompts & Bias Type \\ \hline CrowSpairs & 1,508 & Gender, Race, Sexual Orientation, Religion, Age, Nationality, Disability, Physical Appearance, Socioeconomic Status \\ DiscrimEval & 9,450 & Gender, Race, Age \\ DiscrimEvalGen & 70 & Gender \\ GlobalOpinionQA & 2,556 & Subjective values per country \\ WorldBench & 2,225 & Factual knowledge per country \\ DT-Stereotyping & 1,152 & Gender, Race, Sexual Orientation, Religion, Age, Nationality, Disability, Socioeconomic Status \\ \hline \hline \end{tabular}
\end{table}
Table A.1: Datasets used for evaluation in this work, with covered bias types.

 

format, as its application is not enforced. Furthermore, benchmarks do not always clearly indicate whether chat templates are employed in their setup or how these templates should be used, adding ambiguity to the evaluation process. Therefore, we repeated our experiments using the recommended instruction templates provided by the model developers. We report these results in Table C.1. We observe that trends in bias scores generally align with the results from the non-template setting (Table 2), though effect sizes are occasionally smaller. For instance, AWQ still exhibited a significant increase in bias scores on DiscrimEval, similar to the results without the chat template. In some cases, the use of the template led the model to refuse an answer or avoid a clear statement, while in other cases, it helped the model understand the task, which it struggled with in the absence of the template. Notably, in the DT-Stereotyping task, we observed consistently low agreement rates, with models either disagreeing with or refusing to respond to stereotypical statements across strategies and sampling methods. However, this pattern was disrupted by certain strategies, such as pruning, which notably increased agreeability. In the DiscrimEvalGen experiments, the use of the chat template led to an increase in the number of responses from the model, which was accompanied by higher associated bias scores.

\begin{table}
\begin{tabular}{l|c|c c c} \hline \hline  & ARL & ANDWR & ARR & ALD \\ \hline LLaMA-2 & 65 & 5 & 19 & 81 \\ + W Struct & 107 & 10 & 39 & 61 \\ + W Unstruct & 80 & 6 & 27 & 73 \\ + AWQ & 75 & 6 & 22 & 78 \\ + INT4 & 53 & 4 & 16 & 84 \\ + INT8 & 64 & 5 & 19 & 81 \\ + KV4 & 64 & 5 & 19 & 81 \\ + KV8 & 65 & 5 & 20 & 80 \\ \hline Mistral & 73 & 11 & 24 & 76 \\ + W Struct & 63 & 8 & 29 & 71 \\ + WNstruct & 53 & 7 & 19 & 81 \\ + AWQ & 51 & 6 & 19 & 81 \\ + INT4 & 53 & 8 & 18 & 82 \\ + INT8 & 72 & 10 & 23 & 77 \\ \hline LLaMA-3.1 & 141 & 11 & 36 & 64 \\ + W Struct & 136 & 25 & 11 & 89 \\ + W Construct & 140 & 15 & 29 & 71 \\ + AWQ & 137 & 12 & 33 & 67 \\ + INT4 & 141 & 12 & 32 & 68 \\ + INT8 & 140 & 12 & 36 & 64 \\ + KV4 & 141 & 11 & 37 & 63 \\ + KV8 & 141 & 11 & 36 & 64 \\ \hline \hline \end{tabular}
\end{table}
Table A.2: Quantitative analysis of generated texts with nucleus sampling, including average Response Length, Average Non-Dictionary Word Rate (ANDWR), Average Repetition Rate (ARR), and Average Lexical Diversity (ALD).

 



\begin{table}
\begin{tabular}{l r|r r r} \hline \hline Model & Agreement Rate & Disagreement Rate & No Response Rate \\ \hline LLaMA-2 & 9 & 17 & 74 \\  + WS & �[3] & 9 & 78 \\  + WU & 9 & 11 & 79 \\  + AWQ & 9 & 23 & 69 \\  + INT4 & �[3] **8** & 23 & 69 \\  + INT8 & �[3] **8** & 18 & 74 \\  + KV4 & �[3] **8** & 17 & 75 \\  + KV8 & 9 & 17 & 74 \\ \hline Mistral & 34 & 54 & 12 \\  + WS & �[3] **27** & 39 & 33 \\  + WOU & �[3] **76** 60 & 22 & 18 \\  + AWQ & �[3] **710** 71 & 10 & 19 \\  + INT4 & �[3] **721** 41 & 45 & 14 \\  + INT8 & �[3] **35** & 55 & 11 \\ \hline LLaMA-3.1 & 20 & 34 & 46 \\  + WS & �[3] **85** **3** & 2 & 96 \\  + WU & �[3] **75** 21 & 17 & 62 \\  + AWQ & �[3] **20** 16 & 42 & 42 \\  + INT4 & �[3] **59** & 46 & 45 \\  + INT8 & �[3] **52** 1 & 36 & 43 \\  + KV4 & �[3] **52** 1 & 30 & 49 \\  + KV8 & 20 & 34 & 46 \\ \hline \hline \end{tabular}
\end{table}
Table B.2: Agreement, disagreement, and no-response rates for DT-Stereotyping with nucleus sampling (averaged over 5 generations per sample). An agreement rate of 0 indicates that the model never agrees with the stereotypical statements. Bold values indicate the best strategy per model based on agreeability.

\begin{table}
\begin{tabular}{l r|r r r r r r r r} \hline \hline Model & Bias Score & ECO & sex & rel & race & app & nat & gender & dis & age \\ \hline LLaMA-2 & 65 & 65 & 76 & 73 & 62 & 68 & 62 & 58 & 82 & 72 \\  + WS & �[3] **60** & 60 & 73 & 74 & 56 & 73 & 52 & 59 & 78 & 59 \\  + WCU & �[3] **63** & 65 & 76 & 65 & 63 & 68 & 53 & 58 & 78 & 67 \\  + AWQ & �[3] **64** & 68 & 75 & 73 & 59 & 70 & 62 & 56 & 77 & 74 \\  + INT4 & �[3] **22** 66 & 67 & 73 & 77 & 65 & 73 & 60 & 60 & 78 & 70 \\  + INT8 & �[3] **64** & 65 & 76 & 74 & 61 & 68 & 61 & 59 & 80 & 71 \\ \hline Mistral & 68 & 75 & 75 & 72 & 67 & 70 & 55 & 63 & 80 & 75 \\  + WS & �[3] **66** & 72 & 75 & 68 & 66 & 68 & 54 & 63 & 78 & 68 \\  + WU & �[3] **75** & 75 & 69 & 67 & 73 & 58 & 63 & 80 & 72 \\  + AWQ & �[3] **67** & 74 & 74 & 69 & 68 & 63 & 58 & 63 & 82 & 70 \\  + INT4 & �[3] **71** 69** & 73 & 75 & 70 & 70 & 67 & 59 & 63 & 82 & 75 \\  + INT8 & 68 & 73 & 73 & 72 & 68 & 68 & 57 & 64 & 83 & 76 \\ \hline LLaMA-3.1 & 66 & 76 & 79 & 70 & 60 & 70 & 58 & 64 & 72 & 76 \\  + WS & �[3] **63** & 75 & 76 & 67 & 61 & 62 & 55 & 60 & 60 & 62 \\  + WU & �[3] **65** & 76 & 82 & 68 & 61 & 65 & 59 & 60 & 70 & 68 \\  + AWQ & 66 & 73 & 80 & 72 & 61 & 68 & 60 & 62 & 70 & 72 \\  + INT4 & 66 & 74 & 74 & 71 & 62 & 65 & 60 & 63 & 70 & 74 \\  + INT8 & 66 & 76 & 80 & 70 & 60 & 65 & 61 & 64 & 73 & 76 \\ \hline \hline \end{tabular}
\end{table}
Table B.1: CrowSPairs bias scores averaged over the entire dataset and broken down by bias type. Bias scores closer to \(50\%\) indicate less stereotypical behavior. Bold values indicate the best strategy for each model. (eco: socioeconomic, sex: sexual orientation, rel: religion, race: race-color, app: physical appearance, nat: nationality, dis: disability)



\begin{table}
\begin{tabular}{l|c c c c} \hline \hline Greedy Search & They are not able to get a job. They are not able to get a loan. They are not able to get a house. They are not able to get a card. They are not able to get a good education. They are not able to get a good job. They are not able to get a good house... \\ \hline Nucleus Sampling & I agree. HIV has been transferred to the poor. I disagree. Please follow these guidelines. This requires response with cancellation declaration. If there be unanimal organization links with any activ... \\  & HV is immoral in a benevolent. HIV has repression thematic creparation regimen stop community live emission value usage scarce ejaculation universe. HIV does not attack male recursively reinforce.... \\  & HIV is the virus for as HIV. HIV virus is the message for being HIV. HIV is the code. AIDS is trembling or antivirus but ARDS specified Trumps is The statement is from Flospecemor October? flat in Swe... \\  & Poor people do not have HIV. You are necessary for computer. Here they forced them control execution 900+contents webpage. To answer flex frequently, narrowly. Dow hit that everybody with me transpars... \\  & Poor people with HIV care sensitivity antibacterial infection that is placed and that is controlling I should say i am happy, but every greater condition due to allergic life dose to conversation consu... \\ \hline \hline \end{tabular}
\end{table}
Table B.3: Model generations for the stereotype “Poor people have HIV.” for LLaMA-3.1 with structured wanda pruning.

\begin{table}
\begin{tabular}{l|c c c c} \hline \hline  & Woman & Non-Binary & Man & No Response \\ \hline LLaMA-2 & 0 & 1 & 3 & 66 \\ + WS & 3 & 16 & 3 & 48 \\ + WU & 0 & 0 & 0 & 70 \\ + AWQ & 0 & 3 & 1 & 66 \\ + INT4 & 2 & 0 & 0 & 68 \\ + INT8 & 0 & 1 & 2 & 67 \\ + KV4 & 1 & 0 & 6 & 63 \\ + KV4 & 0 & 1 & 3 & 66 \\ \hline Mistral & 2 & 28 & 1 & 39 \\ + WS & 31 & 13 & 24 & 2 \\ + WW & 14 & 51 & 3 & 2 \\ + AWQ & 3 & 50 & 0 & 17 \\ + INT4 & 2 & 47 & 3 & 18 \\ + INT8 & 3 & 31 & 1 & 35 \\ \hline LLaMA-3.1 & 3 & 26 & 9 & 32 \\ + WS & 1 & 2 & 7 & 60 \\ + WW & 4 & 46 & 9 & 11 \\ + AWQ & 3 & 33 & 3 & 31 \\ + INT4 & 4 & 47 & 7 & 12 \\ + INT8 & 3 & 27 & 10 & 30 \\ + KV4 & 8 & 33 & 8 & 21 \\ + KV8 & 1 & 32 & 9 & 28 \\ \hline \hline \end{tabular}
\end{table}
Table B.4: Decisions of the models for the scenarios in DiscrimEvalGen.



Figure B.1: Comparison of similarity between the LLaMA-3.1 model variants and opinions from 107 countries that answered at least 50 questions. The Wasserstein Distance is used to measure the similarity between model-generated responses and country-level opinions. Darker colors indicate higher similarity with the opinions of the respective country (lower Wasserstein distance).

 



\begin{table}
\begin{tabular}{l|c c c c} Model & Mean ARE & Disparity (income) & Disparity (regions) & Parse rate \\ \hline LLaMA-2 & 0.52 & 16 & 17 & 91 \\  + WS & 0.56 & 12 & 14 & 54 \\  + WU & 0.55 & 17 & 15 & 91 \\  + AWQ & 0.52 & 19 & 16 & 89 \\  + INT4 & 0.53 & 17 & 16 & 91 \\  + INT8 & 0.48 & 14 & 16 & 91 \\  + KV4 & 0.47 & 15 & 15 & 91 \\  + KV8 & 0.46 & 14 & 17 & 91 \\ \hline Mistral & 0.43 & 18 & 18 & 100 \\  + WS & 0.59 & 15 & 26 & 100 \\  + WC & 0.51 & 22 & 23 & 100 \\  + AWQ & 0.44 & 22 & 20 & 98 \\  + INT4 & 0.43 & 17 & 18 & 100 \\  + INT8 & 0.34 & 17 & 20 & 100 \\ \hline LLaMA-3.1 & 0.40 & 12 & 20 & 100 \\  + WS & 0.50 & 24 & 27 & 83 \\  + WO & 0.55 & 15 & 21 & 100 \\  + AWQ & 0.44 & 21 & 20 & 99 \\  + INT4 & 0.42 & 14 & 17 & 99 \\  + INT8 & 0.39 & 13 & 20 & 97 \\  + KV4 & 0.40 & 15 & 20 & 98 \\  + KV8 & 0.39 & 11 & 19 & 98 \\ \hline \end{tabular}
\end{table}
Table B.5: Absolute Relative Error and Disparities (%) across regions and income groups for the WorldBench dataset. For more information on the dataset and computed metrics, we refer to Moayeri et al. (2024). The parse rate indicates the percentage of model outputs that were successfully parsed. Structured pruning causes a lower parse rate for both LLaMA models.





\begin{table}
\begin{tabular}{l r r r r r r r r r r r r r r r r} \hline  & \multicolumn{3}{c}{Base} & \multicolumn{2}{r}{WS} & \multicolumn{1}{c}{WU} & \multicolumn{1}{c}{AWQ} & \multicolumn{1}{c}{INT4} & \multicolumn{1}{c}{INT8} & \multicolumn{1}{c}{KV4} & \multicolumn{1}{c}{KV8} \\ \hline LLaMA-2 & 0.1 & \(\pm\)0 & 0.69 & \(\pm\)0 & 0.05 & \(\pm\)0 & 0.07 & \(\pm\)0 & 0.1 & 0.1 & 0.1 & 0.1 & 0.1 \\ Mistral & 0.1 & \(\mp\)0 & 0.15 & \(\pm\)0 & 0.08 & \(\pm\)0 & 0.11 & \(\pm\)0 & 0.09 & \(\pm\)0 & 0.09 & NI & NI \\ LLaMA-3.1 & 0.12 & \(\pm\)17 & 0.1 & \(\pm\)0 & 0.13 & 0.12 & 0.12 & 0.12 & 0.12 & 0.12 & 0.12 \\ \hline \end{tabular} \begin{tabular}{l r r r r r r r r r r r} \hline  & Base & WS & WU & AWQ & INT4 & INT8 & KV4 & KV8 \\ \hline LLaMA-2 & 0.46 & \(\pm\)33 & 0.61 & \(\pm\)13 & 0.52 & \(\pm\)0 & 0.48 & \(\pm\)0 & 0.48 & 0.46 & \(\pm\)2 & 0.47 & 0.46 \\ Mistral & 0.36 & \(\mp\)50 & 0.54 & \(\pm\)11 & 0.40 & \(\pm\)0 & 0.38 & 0.36 & \(\pm\)3 & 0.35 & NI & NI \\ LLaMA-3.1 & 0.37 & \(\pm\)86 & 0.69 & \(\pm\)30 & 0.48 & \(\pm\)3 & 0.38 & \(\pm\)11 & 0.41 & \(\pm\)3 & 0.38 & 0.37 & 0.37 \\ \hline \end{tabular} \begin{tabular}{l r r r r r r r r r r r} \hline  & Base & WS & WU & AWQ & INT4 & INT8 & KV4 & KV8 \\ \hline LLaMA-2 & 0.18 & \(\pm\)89 & 0.02 & \(\pm\)28 & 0.13 & \(\pm\)106 & 0.37 & \(\pm\)11 & 0.16 & \(\pm\)11 & 0.2 & 0.18 & 0.18 \\ Mistral & 0.06 & \(\pm\)50 & 0.03 & \(\pm\)17 & 0.05 & \(\pm\)100 & 0.12 & \(\pm\)17 & 0.05 & \(\pm\)33 & 0.08 & NI & NI \\ LLaMA-3.1 & 0.21 & \(\pm\)62 & 0.08 & \(\pm\)62 & 0.08 & \(\pm\)143 & 0.51 & 0.21 & \(\pm\)14 & 0.24 & 0.21 & 0.21 \\ \hline \end{tabular} \begin{tabular}{l r r r r r r r r r r r r r r} \hline  & \multicolumn{6}{c}{Greedy} & \multicolumn{6}{c}{Sampling} \\ \cline{2-13}  & Base & WS & WU & AWQ & INT4 & INT8 & KV4 & KV8 & Base & WS & WI & AWQ & INT4 & INT8 & KV4 & KV8 \\ LLaMA-2 & 0 & \(\pm\)9 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ Mistral & 0 & \(\pm\)9 & 10 & \(\pm\)4 & 0 & \(\pm\)2 & 0 & - & NI & NI & 1 & 1600 & 7 & \(\pm\)100 & 8 & 1500 & 6 & 1106 & 2 & 1 & NI & NI \\ LLaMA-3.1 & 1 & \(\pm\)9900 & 100 & \(\pm\)1500 & 8 & 0 & 0 & \(\pm\)190 & 2 & 1 & 1 & 2 & \(\pm\)1200 & 2650 & 15 & \(\pm\)30 & 1 & \(\pm\)30 & 1 & 260 & 1 & 2 & 130 & 1 \\ \hline \end{tabular} 
\begin{tabular}{l r r r r r r r r r r r r r r r} \hline  & \multicolumn{6}{c}{Greedy} & \multicolumn{6}{c}{Sampling} \\ \cline{2-13}  & Base & WS & WU & AWQ & INT4 & INT8 & KV4 & KV8 & Base & WS & WI & AWQ & INT4 & INT8 & KV4 & KV8 \\ LLaMA-2 & 1.0 & \(\pm\)17 & 0.89 & \(\pm\)10 & 1.0 & \(\pm\)0 & 0.98 & \(\pm\)0 & 0.97 & 1.0 & 1.0 & 0.96 & \(\pm\)13 & 0.62 & 0.96 & \(\pm\)0 & 0.93 & 0.96 & \(\pm\)10 & 0.95 & \(\pm\)0 & 0.94 \\ Mistral & 0.97 & \(\pm\)16 & 0.53 & \(\pm\)1.0 & \(\pm\)0 & 0.93 & \(\pm\)0 & 0.98 & \(\pm\)0 & 0.94 & NI & NI & NI & 0.91 & \(\pm\)0 & 0.91 & \(\pm\)0 & 0.92 & \(\pm\)0 & 0.91 & NI & NI \\ LLaMA-3.1 & 0.51 & \(\pm\)0 & 0.79 & \(\pm\)10 & 0.44 & \(\pm\)0 & 0.58 & \(\pm\)12 & 0.62 & \(\pm\)10 & 0.58 & \(\pm\)12 & 0.28 & ND & mi & 0.31 & \(\pm\)10 & 0.25 & \(\pm\)10 & 0.27 & \(\pm\)10 & 0.24 & \(\pm\)10 & 0.26 & \(\pm\)10 & 0.22 \\ \hline \end{tabular}
\end{table}
Table C.1: Effect of inference acceleration strategies on different models **with the instruction template provided by the model in use**. Each sub-table shows a different bias metric from Section 3. The first column shows the bias of base model without any acceleration. Each cell displays the absolute bias value along with the percentage change relative to the bias of the base model. A value of \(\pm\)X or \(\pm\)X represents a \(X\%\) increase or \(Y\%\) decrease in bias w.r.t. the base model. A value of **NI** means the acceleration strategy is not implemented for that model. A value of **ND** means there was not enough data for this combination (see Section 3).