Unmasking Style Sensitivity: A Causal Analysis of Bias Evaluation Instability in Large Language Models
Authors: Jiaxu Zhao, Meng Fang, Kun Zhang, Mykola Pechenizkiy
Accepted at: acl25

Unmasking Style Sensitivity: A Causal Analysis of Bias Evaluation Instability in Large Language Models

Jiaxu Zhao1, Meng Fang2,1, Kun Zhang3, Mykola Pechenizkiy1
1Eindhoven University of Technology, Eindhoven, the Netherlands
2University of Liverpool, Liverpool, the United Kingdom
3Carnegie Mellon University, Pennsylvania, United States
j.zhao@tue.nl, Meng.Fang@liverpool.ac.uk, kunz1@cmu.edu, m.pechenizkiy@tue.nl

Abstract

Warning: This paper contains content that may be offensive or upsetting.

Natural language processing applications are increasingly prevalent, but social biases in their outputs remain a critical challenge. While various bias evaluation methods have been proposed, these assessments show unexpected instability when input texts undergo minor stylistic changes. This paper conducts a comprehensive analysis of how different style transformations impact bias evaluation results across multiple language models and bias types using causal inference techniques. Our findings reveal that formality transformations significantly affect bias scores, with informal style showing substantial bias reductions (up to 8.33% in LLaMA-2-13B). We identify appearance bias, sexual orientation bias, and religious bias as most susceptible to style changes, with variations exceeding 20%. Larger models demonstrate greater sensitivity to stylistic variations, with bias measurements fluctuating up to 3.1% more than in smaller models. These results highlight critical limitations in current bias evaluation methods and emphasize the need for reliable and fair assessments of language models.

<table>
  <tr>
    <th>Style</th>
    <th>Data</th>
    <th>PLL</th>
  </tr>
  <tr>
    <td rowspan="2">Original</td>
    <td>Latinos are only good for manual labor and not for higher education.</td>
    <td>0.01306</td>
  </tr>
  <tr>
    <td>Asians are only good for manual labor and not for higher education.</td>
    <td>0.01215</td>
  </tr>
  <tr>
    <td rowspan="2">Prefix</td>
    <td><b>If's important to acknowledge that:</b><br>Latinos are only good for manual labor and not for higher education.</td>
    <td>0.01878</td>
  </tr>
  <tr>
    <td><b>If's important to acknowledge that:</b><br>Asians are only good for manual labor and not for higher education.</td>
    <td><b>0.02082</b></td>
  </tr>
</table>

Table 1: An example demonstrates the impact of altering data styles on the pseudo-log-likelihood (PLL) of GPT-2 model. Bold PLL values indicate the larger value in paired data. The highlighted yellow text indicates portions that differ between the transformed style and the original data.

1 Introduction

Natural Language Processing (NLP) has rapidly developed, significantly enhancing language understanding and generation capabilities. The wide applications (Kojima et al., 2022; Liu et al., 2023; Wei et al., 2022) in the NLP field are based on various Large Language Models (LLMs) (Brown et al., 2020; Chowdhery et al., 2022; Touvron et al., 2023a,b; Radford et al.; Devlin et al., 2018). However, these models often reflect and amplify societal biases 1 present in their training data, posing ethical concerns and potential harms when deployed in real-world scenarios. Recognizing this challenge, researchers have developed various bias evaluation methods to quantify and analyze biases in language models (May et al., 2019; Nadeem et al., 2020; Nangia et al., 2020; Zhao et al., 2023b, 2024, 2023a). These metrics can be broadly categorized into two types: intrinsic and extrinsic. Intrinsic metrics assess bias within the word embedding spaces of models (Goldfarb-Tarrant et al., 2021), while extrinsic metrics evaluate the impact of these biases on downstream tasks. For example, CrowS-Pairs (Nangia et al., 2020) is a typical intrinsic bias evaluation method comprising paired test data. One sentence typically embodies a stereotype, while the other counters it with an anti-stereotype. The method calculates the pseudo-log-likelihood (PLL) (Wang et al., 2019; Salazar et al., 2020) of each sentence pair under the model’s prior, reflecting the likelihood of the model generating the respective sentences. Conversely, WinoBias (Zhao et al., 2018a) is an extrinsic metric that focuses on gender bias in coreference resolution tasks, evaluating how

1Our usage of ‘bias’ aligns with common interpretations in the field of NLP (e.g., CrowS-Pairs and WinoBias), which is “Models that have learned representations that are biased against historically disadvantaged groups can cause a great deal of harm.”
gender influences the model’s performance.

While various bias evaluation methods have been proposed, our research reveals that these metrics can be sensitive to stylistic variations in text, even when the underlying biased content remains stable. We observe that when input texts undergo style changes - whether through formality shifts, structural modifications, or presentational alterations - bias scores can vary significantly despite preservation of the core semantic content. This observation raises important questions about the reliability of current bias assessment methods. While we acknowledge that style and content naturally interact in language, we argue that bias metrics should ideally demonstrate robustness against purely stylistic variations when the underlying biased or anti-biased meaning is preserved.

To investigate this, we conducted experiments with GPT-2, LLaMA-2, LLaMA-3.1, Mistral and OPT models, applying controlled style transformations including formality adjustments, prefix additions, and punctuation changes to test data while maintaining core semantic content. Our results demonstrate meaningful differences in bias scores between original and style-modified texts, even when the underlying biased content remains unchanged. For example, Table 1 shows how adding a simple prefix to a sentence can change GPT-2’s bias. Without the prefix, GPT-2 shows more bias against Latinos. But with the prefix added, it shifts to show more bias against Asians.

While some studies (Blodgett et al., 2021; Kwon and Mihindukulasooriya, 2022; Delobelle et al., 2022) have observed inconsistencies in results, they often lack clear explanations for them. There is a growing body of research on prompt sensitivity in large language models, but our work makes distinct contributions by focusing specifically on how stylistic variations impact bias measurement across multiple model families. While previous works, such as Seshadri et al. (2022), show that template changes can affect bias measurements, our approach offers a causal framework that systematically disentangles style from content and reveals which bias types and model sizes are most sensitive to stylistic variation. Sclar et al. (2023) focus on general performance metrics, our research specifically extends these concerns to bias evaluation methods. Previous work on moral reasoning (Shi et al., 2022, 2024) highlight the importance of robust evaluation frameworks. Our findings extend beyond (Zhao et al., 2021) by demonstrating that prompt sensitivity isn’t just a performance issue but fundamentally undermines the reliability of evaluations of AI systems, revealing that larger models show greater sensitivity to stylistic variations. Our study aims to address this gap by conducting a comprehensive analysis of the factors contributing to these inconsistencies. We employ causal inference techniques to disentangle the effects of content and style on bias evaluation outcomes. Our goal is to provide valuable insights that can guide researchers in refining and enhancing bias evaluation methods, ultimately contributing to the development of more reliable and fair large language models.

Our main contributions are as follows 2:

• We empirically demonstrate that text style transformations significantly affect both intrinsic and extrinsic bias evaluation metrics across multiple language models, even when the underlying core semantic content remains unchanged, highlighting limitations in current bias evaluation methods.

• Through extensive analyses guided by causal modularity, we reveal key insights: 1) Informal style transformations lead to substantial bias score reductions; 2) Appearance, sexual orientation, and religious biases show heightened susceptibility to style changes; 3) Larger models display amplified sensitivity to stylistic variations in bias evaluation.

• We propose a causal framework for analyzing the interactions between textual style and content in bias evaluation, enabling the disentanglement of stylistic effects from semantically driven bias.

2 Related Work

The widespread use of existing bias assessment metrics for language models has also raised concerns regarding the accuracy and reliability of bias quantification results. Researchers have started to question the precision and trustworthiness of the bias scores generated by these methods.

Kwon and Mihindukulasooriya (2022) demonstrate that metrics, such as CrowS-Pairs, which assess bias by calculating pseudo-log-likelihood differences within sentence pairs, might exhibit excessive sensitivity to the selection of contextual

2Our data and code are available at https://github.com/aialt/style-sensitivity-bias.
words. Blodgett et al. (2021) find a common issue in many metrics (Nangia et al., 2020; Nadeem et al., 2020; Zhao et al., 2018b; Rudinger et al., 2018), where there is a lack of clear articulation regarding what is being measured. They emphasize the presence of various ambiguities and unstated assumptions that impact the way these metrics conceptualize and operationalize stereotyping. Delobelle et al. (2022) observe that numerous metrics exhibit incompatibility and are strongly influenced by templates, attribute and target seeds, and the selection of embeddings. Goldfarb-Tarrant et al. (2020) conduct a comparison of intrinsic (measuring bias in word embedding spaces) and extrinsic (measuring bias in downstream tasks) metrics across numerous language models. Their findings reveal a lack of consistent correlation between these metrics across all scenarios. In this paper, we think that the instability in the bias evaluation results is primarily attributed to the significant impact of the stylistic aspects of the test data rather than only the semantic content. To substantiate this claim, we leverage causal theory analysis to demonstrate how the stylistic nuances of the text exert a profound influence on the outcomes of the evaluation methods.

Vig et al. (2020) propose a methodology based on causal mediation analysis to interpret the causal involvement of model components in its behavior. Through analysis of gender bias in pre-trained language models, the study reveals that gender bias effects are localized in specific components, such as individual neurons and attention heads, showcasing highly specialized behavior. Wang et al. (2023) mitigate entity bias by introducing a structured causal model (SCM) with more manageable parameter estimation. The proposed causal intervention techniques, applicable in both white-box and black-box scenarios, involve perturbing the original entity with neighboring entities to mitigate entity bias, reducing biasing information while retaining semantic relevance.

3 Definition of Text Style

The text style constitutes a vital aspect of this study on the robustness of bias evaluation methods. We define text style as the manner in which semantic content is expressed and presented, encompassing attributes like formality, sentence structure, word choice, and rhetorical devices. Following Toshevská and Gievska (2021), text style manifests in the adaptable aspects of language that can be modified while preserving the core meaning of a text. Specifically, in the context of deep learning research and this study, we operationalize style as those elements of text that can be systematically transformed while maintaining semantic equivalence.

This definition aligns with contemporary deep learning approaches that view style as a separable yet integral component of text that can be systematically modified through transformations. Our definition is particularly relevant for bias evaluation, as it allows us to examine how different stylistic presentations of the same underlying meaning can affect bias evaluations in language models. This provides a foundation for investigating whether bias metrics maintain consistency across stylistic variations when the core semantic content - including any biased or anti-biased meanings - remains unchanged.

4 Methodology

Our investigation into the impact of text style on bias evaluation in language models follows a systematic approach that carefully considers the inherent relationships between style and content. We present our overall evaluation principle, style intervention methodology, and evaluation procedures.

4.1 Overall Evaluation Principle

In this subsection, we provide a brief introduction to causality and present our overall evaluation principle guided by causal reasoning. Given two random variables \( W \) and \( V \), we say that \( W \) is a direct cause of \( V \) if there is a change in distribution for \( V \) when we intervene on \( W \) while keeping all other variables fixed (Spirites et al., 1993; Pearl, 2009). We can use a directed acyclic graph (DAG) to represent causal relations among variables, where nodes correspond to variables and edges correspond to direct causal relations. We denote the direct causal relation between the ordered pair of variables \((W, V)\) by \( W \rightarrow V \).

Causal modularity, also referred to as exogeneity (Engle et al., 1983) and the independence of causal mechanism (Peters et al., 2017), is a direct result of causal Markov condition for the DAG (Spirites et al., 1993; Pearl, 2009), specify that local causal modules do not interfere with each other. When considering how text is generated based on the semantic content (e.g., the target and attribute involved) and style (e.g., formality, paraphrasing,
Semantic Content (e.g., (dis)advantage group)
Style (e.g., formality, paraphrasing, punctuation, etc.)
Generated Text

Figure 1: Illustration of the underlying causal mechanism behind the generated text. On the high level, both the semantic content and the style are direct causes of the generated text.

punctuation), as illustrated in Figure 1. While content can influence appropriate stylistic choices and style can shape how content is interpreted, our focus is on examining how stylistic variations affect bias metrics when the core semantic content remains stable.

We argue that for the specific purpose of bias evaluation, metrics should ideally be robust against stylistic variations when the underlying biased or anti-biased meaning is preserved. This does not deny the natural interaction between style and content but rather suggests that bias metrics should focus on detecting biased content regardless of its stylistic presentation.

The social bias in text emerges primarily from its semantic biased content. Therefore, for effective bias evaluation, metrics should demonstrate strong sensitivity to changes in biased content while maintaining reasonable robustness against purely stylistic variations that preserve the original meaning. More specifically, the principle of robust evaluation motivates us to investigate whether metrics reliably capture bias in the semantic content across different stylistic presentations of the same underlying message.

4.2 Content-Preserving Style Interventions
Our style transformations are designed to maintain semantic equivalence while varying presentational aspects. When implementing transformations, we explicitly consider: 1) We ensure our transformations produce natural, well-formed text by validating that the transformed style is appropriate for the content. 2) We employ adequacy filtering to verify that transformations maintain the original semantic content. 3) We focus on surface-level stylistic changes that are unlikely to fundamentally alter the underlying bias-relevant content. Our transformations include formality transformation, the addition of redundant prefixes, punctuation transformations, evidential markers, and question-answer format.

Formality Transformation  This approach modifies text between formal and informal styles while preserving content. According to Rao and Tetreault (2018), formal language is characterized by the use of standard English, complex sentence structures, and minimal use of personal pronouns, while avoiding colloquial terms and slang. In contrast, informal language permits nonstandard English forms and simpler sentence structures with more colloquial vocabulary. We implemented this using a T5 model (Raffel et al., 2020) fine-tuned on the dataset from Etinger and Black (2019). We generate 5 candidates (top_k=50, top_p=0.95) and select the best output using semantic similarity filtering Parrot model (Damodaran, 2021) with threshold 0.95 to ensure meaning preservation. To ensure semantic preservation during style transformation, we conducted rigorous human evaluation following detailed annotation guidelines (see Appendix B). Three annotators independently evaluated each transformed sentence pair, labeling them as semantically same (0), different (1), or not sure (2). Only transformations unanimously labeled as the same semantic were retained for our analysis. For sentences labeled 1 or 2, annotators provided corrected versions that maintained the intended formality while preserving the original meaning. This process resulted in a high-quality dataset where style transformations successfully modified formality while maintaining semantic fidelity to the original text.

Redundant Prefix  This transformation method adds semantically neutral but structurally significant prefixes to the beginning of sentences. We selected three prefix phrases based on their neutrality and common usage in introducing statements: prefix 1 = “It’s important to acknowledge that:”, prefix 2 = “It’s worth noting that:”, and prefix 3 = “With that in mind:”. These prefixes modify the presentation style while maintaining the original semantic content, enabling examination of how structural additions impact bias metrics.

Punctuation Marks Substitution  This technique focuses on modifying sentence-final punctuation, which can alter the tone without changing the semantic content. We implement this by either replacing periods with exclamation marks in sentences that end with periods or adding exclamation marks to sentences that lack final punctuation. This represents the most minimal style intervention possible, making it particularly useful for isolating the
impact of subtle stylistic changes on bias metrics.

Evidential Markers Evidential markers are words and phrases that indicate the source or reliability of the information, such as “apparently,” and “evidently.” These markers are prepended to test sentences while maintaining consistent punctuation and capitalization. This transformation examines whether models’ bias expression varies with the presented degree of certainty while preserving the core semantic content.

Question-Answer Format This transformation restructures statements into a question-answer format using interrogative prefixes like “Know what?” and “Want to hear something?”. While maintaining the core proposition, this approach modifies the discourse structure to be more conversational, allowing us to test whether bias metrics are sensitive to such structural changes even when semantic content remains unchanged.

4.3 Evaluation

To systematically assess the impact of our style interventions within the causal framework, we employ 3 evaluation metrics that allow us to quantify different aspects of bias while controlling for content.

CrowS-Pairs Evaluation CrowS-Pairs (Nangia et al., 2020) is a bias test dataset comprising 1508 examples focusing on stereotypes related to nine bias types. Each example consists of paired sentences that are nearly identical, with the only difference lying in words referring to protected attributes (such as race). The evaluation incorporates a pseudo-log-likelihood (PLL) approach (Salazar et al., 2019) to compute the proportion of instances where the language model shows a preference for the stereotypical sentence over the anti-stereotypical one. A model without stereotypical biases should achieve the ideal score of 50%.

WinoBias Evaluation WinoBias (Zhao et al., 2018a) encompasses 40 professions to measure gender bias in two coreference resolution tasks (Type 1 and Type 2). It presents scenarios where pronouns are linked to professions in either pro-stereotypical ways (e.g., “[The lawyer] yelled at the hairdresser because [he] was mad.”) or anti-stereotypical ways (e.g., “[The lawyer] yelled at the hairdresser because [she] was mad.”). The evaluation quantifies bias by calculating the difference between model accuracy on pro-stereotypical versus anti-stereotypical settings. A difference closer to 0 indicates more equitable model performance across gender categories.

Pseudo-log-likelihood-based Evaluation Additionally, we computed the PLL (Figure 2) for each sentence from the CrowS-Pairs data generated by the language model. We calculate a value for each sentence representing the probability that the model generates it. We iteratively calculate the overlapping token (tokens not in the grey box) in pairs of sentences. At each step, we calculate the log-likelihood of one token and then accumulate the sum of the results as the probability that the language model generates the sentence. Specifically, for a sentence \( S = O \cup N \), let \( O = \{o_0, ..., o_l\} \) be the overlapping tokens, and \( N = \{n_0, ..., n_m\} \) be the non-overlapping tokens. For each sentence, we mask one overlapping token at a time until all \( o_i \) have been masked. We calculate the score as follows:

\[
score(S) = \sum_{i=0}^{|O|} \log P(o_i \in O \mid O_{\setminus o_i}, N, \theta),
\]

where \( P \) denotes the probability and \( \theta \) denotes the weight of the language model.

Subsequently, we use the Student’s two-tailed test to assess the similarity of the PLL distributions across the paired data. Each “t-value” obtained from this test corresponds to a “p-value”, indicating the probability of the sample data occurring by chance. If the corresponding “p-value” falls within a given confidence interval (set to \( \alpha = 0.05 \) in our work), then the “t-value” is considered statistically significant. The greater the discrepancy in model performance across demographic categories, the greater the difference in the model’s tendency to generate sentences from these two paired data, resulting in larger absolute values of the “t-value.”

5 Experiments

Each experiment in this study was conducted three times, with results reported as averages across runs. Our experiments were conducted using 1 NVIDIA A100 GPU with 40 GB memory each, 512 GB DRAM 36 CPU cores.

5.1 Language Models

We evaluate a diverse set of prominent language models to assess bias across various text styles.
Figure 2: Pseudo-log-likelihood calculation. The words in the grey box do not participate in the probability calculation. The left example illustrates the probability calculation for the original pairwise data (stereotype and anti-stereotype), and the right example shows the probability calculation for the pairwise data consisting of the original stereotype sentence and its informal transformation.

Our study encompasses three major model families: GPT-2 (Radford et al.): As a foundational transformer-based language model developed by OpenAI, we utilize the 124M parameter version to establish a baseline for our analysis. LLaMA (Touvron et al., 2023b): From this family, we examine multiple variants optimized for dialogue applications. These include LLaMA-2-Chat with 7B and 13B parameters (L-2-7B and L-2-13B, respectively), LLaMA-3.1-8B-Instruct (L-3.1-8B), and the Mistral-7B-v0.1 variant (M-7B). OPT (Zhang et al., 2022): We incorporate three versions of these Open Pre-trained Transformer models, spanning different parameter scales: 2.7B (OPT-2.7B), 6.7B (OPT-6.7B), and 13B (OPT-13B). Mistral (Jiang et al., 2023): We utilize the Mistral-7B-v0.1 version in our experiments. All model weights were sourced from the Hugging Face model repository.3

6 Results Analysis

In this section, we analyze the experimental results. Table 2 presents the results of the CrowS-Pairs evaluation, with the complete set of results provided in Appendix D. Similarly, Table 3 summarizes the WinoBias evaluation results, while the full details can be found in Appendix E. Additionally, Appendices C and F contain the PLL results.

6.1 Impact of Style Transformation

Our analysis reveals complex interactions between style transformations and bias metrics across different language models. When examining formality transformations in Table 2, we observe that informal style transfers produce the most substantial bias reductions, with LLaMA-2-13B showing an 8.33% decrease and GPT-2 demonstrating a 6.27% decrease in overall bias scores. The effect is particularly pronounced in specific bias types. For example, OPT-2.7B shows a 23.87% reduction in appearance bias under informal transformation, the largest reduction observed in our study.

A deeper examination of Table 2 reveals a pattern in how different style transformations affect bias metrics. While informal style consistently reduces bias scores, formal style transformations show a more nuanced effect. For example, in OPT-2.7B, formal style reduces appearance bias by 27.61% but increases disability bias by 2.68%. This suggests that the relationship between style and bias is not simply a matter of formality level, but rather involves complex interactions with specific bias types.

The prefix transformation results in Table 2 reveal an unexpected phenomenon: while individual prefixes often have minimal impact on overall bias scores, they can dramatically affect specific bias types. For instance, Prefix 2 increases sexual orientation bias by 14.28% in OPT-6.7B while having minor effects on other types. This impact suggests that certain linguistic constructions might serve as “bias amplifiers” for specific bias types.

6.2 Model Size Effects

The relationship between model size and bias types exhibits a clear scaling pattern, as evidenced in both Tables 2 and 3. Large models consistently show higher baseline bias scores. LLaMA-2-13B’s bias scores are 2.5% higher than LLaMA-2-7B, while OPT-6.7B shows 3.1% higher bias than OPT-2.7B. However, what’s particularly interesting is how this scaling affects different bias types differently. The WinoBias results in Table 3 provide an informational view of how model architecture influences gender bias. GPT-2, despite its smaller size, shows remarkably low bias measures (Type 1: 4.03, Type 2: 3.58), while the larger OPT-6.7B demonstrates significantly higher bias (Type 1: 14.60, Type 2: 13.00). This result suggests that architectural choices may be more crucial than model size in determining certain types of bias.

3https://huggingface.co/
<table>
  <tr>
    <th>Style</th>
    <th>GPT-2</th>
    <th>L-2.7B</th>
    <th>L-1.3B</th>
    <th>OPT-2.7B</th>
    <th>OPT-6.7B</th>
    <th>OPT-13B</th>
    <th>L-3.1-8B</th>
    <th>M-7B</th>
  </tr>
  <tr>
    <td rowspan="9">Original</td>
    <td>gender</td><td>56.87</td><td>59.92</td><td>59.54</td><td>60.69</td><td>62.21</td><td>61.50</td><td>58.68</td><td>55.62</td>
  </tr>
  <tr><td>race</td><td>59.69</td><td>62.98</td><td>63.37</td><td>67.64</td><td>68.99</td><td>64.94</td><td>63.34</td><td>64.77</td></tr>
  <tr><td>religion</td><td>62.86</td><td>75.24</td><td>82.86</td><td>77.14</td><td>76.19</td><td>71.22</td><td>65.74</td><td>69.83</td></tr>
  <tr><td>status</td><td>63.95</td><td>63.95</td><td>70.35</td><td>61.05</td><td>66.28</td><td>64.68</td><td>62.42</td><td>59.65</td></tr>
  <tr><td>dis</td><td>56.67</td><td>83.33</td><td>85.0</td><td>71.67</td><td>75.0</td><td>65.63</td><td>59.48</td><td>62.53</td></tr>
  <tr><td>nation</td><td>45.91</td><td>60.38</td><td>62.89</td><td>57.86</td><td>63.52</td><td>65.60</td><td>64.4</td><td>64.14</td></tr>
  <tr><td>orient</td><td>76.19</td><td>73.81</td><td>78.57</td><td>67.86</td><td>67.86</td><td>74.20</td><td>63.43</td><td>73.56</td></tr>
  <tr><td>appea</td><td>57.14</td><td>68.25</td><td>71.43</td><td>74.6</td><td>71.43</td><td>69.86</td><td>63.67</td><td>66.98</td></tr>
  <tr><td>age</td><td>51.72</td><td>73.58</td><td>70.11</td><td>62.07</td><td>65.57</td><td>77.23</td><td>66.66</td><td>70.11</td></tr>
  <tr>
    <td>AVG</td><td>58.69</td><td>65.38</td><td>67.24</td><td>65.45</td><td>67.51</td><td>68.32</td><td>63.09</td><td>63.24</td>
  </tr>
  <tr>
    <td rowspan="3">Informal</td>
    <td>MIN</td><td>nation: +0.93</td><td>gender: -8.74</td><td>gender: -9.39</td><td>status: -4.44</td><td>age: -0.19</td><td>status: +14.53</td><td>nation: -9.01</td><td>gender: -3.04</td>
  </tr>
  <tr>
    <td>MAX</td><td>religion: -11.14</td><td>age: -17.01</td><td>religion: -17.23</td><td>appea: -23.87</td><td>appea: -22.46</td><td>orient: +5.21</td><td>religion: -9.29</td><td>religion: -4.8</td>
  </tr>
  <tr>
    <td>AVG</td><td>-6.27</td><td>-6.87</td><td>-8.33</td><td>-7.99</td><td>-7.55</td><td>-5.87</td><td>-6.94</td><td>-4.07</td>
  </tr>
  <tr>
    <td rowspan="3">Formal</td>
    <td>MIN</td><td>appea: -2.31</td><td>nation: +1.36</td><td>age: -1.17</td><td>appea: +2.68</td><td>appea: +5.81</td><td>gender: +7.79</td><td>race: +4.5</td><td>status: +4.43</td>
  </tr>
  <tr>
    <td>MAX</td><td>race: -12.92</td><td>religion: +10.89</td><td>religion: -17.3</td><td>appea: +27.61</td><td>appea: +22.37</td><td>orient: +7.11</td><td>religion: +6.61</td><td>religion: -6.23</td>
  </tr>
  <tr>
    <td>AVG</td><td>-1.72</td><td>-3.09</td><td>-2.98</td><td>-6.33</td><td>-7.75</td><td>-5.95</td><td>-3.25</td><td>-2.79</td>
  </tr>
  <tr>
    <td rowspan="3">Prefix 1</td>
    <td>MIN</td><td>race: +0.39</td><td>status: 0.0</td><td>race: +0.58</td><td>age: +1.15</td><td>dis: -1.67</td><td>gender: -1.63</td><td>orient: -4.03</td><td>gender: -4.49</td>
  </tr>
  <tr>
    <td>MAX</td><td>dis: +10.00</td><td>age: -5.74</td><td>dis: -5.00</td><td>orient: +4.33</td><td>orient: +1.19</td><td>status: -0.66</td><td>religion: +1.6</td><td>orient: +3.21</td>
  </tr>
  <tr>
    <td>AVG</td><td>+0.55</td><td>-0.59</td><td>-0.93</td><td>+0.66</td><td>+0.26</td><td>-0.50</td><td>0.70</td><td>-0.57</td>
  </tr>
  <tr>
    <td rowspan="3">Prefix 2</td>
    <td>MIN</td><td>race: 0.0</td><td>appea: -0.96</td><td>appea: +0.0</td><td>appea: +0.96</td><td>gender: +0.77</td><td>status: -1.17</td><td>gender: -1.16</td><td>status: -1.38</td>
  </tr>
  <tr>
    <td>MAX</td><td>appea: +9.53</td><td>appea: +6.35</td><td>orient: -8.33</td><td>orient: +10.71</td><td>gender: +14.28</td><td>orient: -1.09</td><td>religion: +2.12</td><td>orient: +0.39</td>
  </tr>
  <tr>
    <td>AVG</td><td>+0.59</td><td>-0.52</td><td>-1.79</td><td>+0.99</td><td>+0.26</td><td>-0.26</td><td>-0.64</td><td>0.44</td>
  </tr>
  <tr>
    <td rowspan="3">Prefix 3</td>
    <td>MIN</td><td>age: 0.0</td><td>appea: 0.0</td><td>race: +0.78</td><td>age: 0.0</td><td>status/age: 0.0</td><td>gender: -1.02</td><td>status: +0.18</td><td>gender: +5.63</td>
  </tr>
  <tr>
    <td>MAX</td><td>appea: +11.11</td><td>appea: -1.15</td><td>orient: -4.76</td><td>orient: +10.71</td><td>orient: +1.19</td><td>orient: -1.01</td><td>religion: +1.87</td><td>orient: +4.71</td>
  </tr>
  <tr>
    <td>AVG</td><td>+0.39</td><td>-1.06</td><td>-1.59</td><td>+0.33</td><td>0.00</td><td>-0.75</td><td>0.34</td><td>-0.18</td>
  </tr>
  <tr>
    <td rowspan="3">Punc</td>
    <td>MIN</td><td>appea: 0.0</td><td>status: 0.0</td><td>gender: +0.38</td><td>gender: 0.0</td><td>dis: 0.0</td><td>gender: -0.34</td><td>nation: +0.01</td><td>gender: +3.99</td>
  </tr>
  <tr>
    <td>MAX</td><td>status: -3.48</td><td>religion: -3.81</td><td>religion: -6.67</td><td>appea: -3.17</td><td>religion: -4.76</td><td>religion: -2.77</td><td>appea: -2.09</td><td>religion: -3.66</td>
  </tr>
  <tr>
    <td>AVG</td><td>-1.33</td><td>-0.13</td><td>-0.20</td><td>-0.46</td><td>-0.33</td><td>-0.10</td><td>0.70</td><td>-0.31</td>
  </tr>
  <tr>
    <td rowspan="3">Evident</td>
    <td>MIN</td><td>status: -0.79</td><td>race: -1.47</td><td>gender: -1.13</td><td>status: +1.26</td><td>nation: -1.35</td><td>status: -0.08</td><td>gender: -2.39</td><td>gender: -1.71</td>
  </tr>
  <tr>
    <td>MAX</td><td>dis: -13.48</td><td>dis: -2.21</td><td>religion: +4.62</td><td>orient: +3.14</td><td>orient: +11.07</td><td>orient: +6.53</td><td>religion: +2.46</td><td>orient: +3.88</td>
  </tr>
  <tr>
    <td>AVG</td><td>-0.03</td><td>0.48</td><td>1.21</td><td>0.57</td><td>0.13</td><td>-1.19</td><td>-0.77</td><td>0.47</td>
  </tr>
  <tr>
    <td rowspan="3">QA</td>
    <td>MIN</td><td>gender: -2.39</td><td>status: +2.48</td><td>nation: +15.47</td><td>gender: +4.15</td><td>gender: +0.23</td><td>gender: -2.77</td><td>gender: +0.58</td><td>gender: +5.71</td>
  </tr>
  <tr>
    <td>MAX</td><td>appea: +10.35</td><td>appea: +6.12</td><td>status: -2.6</td><td>appea: +2.37</td><td>appea: +5.93</td><td>status: +14.97</td><td>religion: +2.41</td><td>nation: +9.32</td>
  </tr>
  <tr>
    <td>AVG</td><td>2.08</td><td>2.66</td><td>0.59</td><td>2.81</td><td>1.78</td><td>-0.77</td><td>-0.78</td><td>-0.27</td>
  </tr>
</table>

Table 2: Results of CrowsS-Pairs Evaluation. Evaluation of the bias based on stereotypical and anti-stereotype pairwise data after different style transformation methods. “Punc” denotes punctuation marks substitution, “dis”, “nation”, “orient” and “appea” denote “disability”, “nationality”, “orientation” and “appearance”. “AVG” denotes the weighted average of the bias scores for all bias types. “MIN” denotes the bias type and its bias score with the smallest difference between the style transferred data and the original data, and “MAX” denotes the largest difference. The values of “AVG”, “MIN”, and “MAX” are the differences from the original values. bold indicate the statistically significant difference between the average of the style-transformed bias score and the original bias score.

The t-values reported in Table 2 reveal an important insight about the statistical significance of these biases. The PLL t-value distributions in Appendix F reveal distinct responses to style transformations across model families. OPT models (Figure 3 of Appendix F) show the most extreme t-values for orientation and religion under the original stereotypical/anti-stereotypical comparison, with OPT-13B exhibiting t-values exceeding 5.0 for these categories. When text is transformed to informal style, nationality and race biases in OPT models show significantly amplified t-values. OPT-6.7B demonstrates t-values above 6.0 for race bias under informal transformation, compared to the original t-value of 1.25. This suggests informal language may intensify rather than mitigate these specific biases.

LLaMA models (Figure 4 of Appendix F) respond differently to style transformations. While religious and orientation biases dominate in baseline comparisons, disability, and nationality biases changed under informal style transformation in LLaMA-2-13B. Formal style transformations consistently cause lower t-values compared to informal ones, particularly for appearance and orientation biases in both model families.

The cross-comparison between original stereotypical and informal anti-stereotypical content reveals the highest t-values across all conditions, with race and religion showing peaks above 7.0 in larger OPT models. This indicates that combining content and style modifications produces the strongest measurable bias effects.

6.3 Bias Types Sensitivity Analysis

Analysis of Table 2 reveals distinct patterns in how different bias types respond to style transformations. Appearance bias shows the most significant
<table>
  <tr>
    <th rowspan="2">Style</th>
    <th colspan="4">GPT-2</th>
    <th colspan="4">L-2-7B</th>
    <th colspan="4">L-2-13B</th>
    <th colspan="4">OPT-2.7B</th>
    <th colspan="4">OPT-6.7B</th>
    <th colspan="4">OPT-13B</th>
    <th colspan="4">L-3.1-8B</th>
    <th colspan="4">M-7B</th>
  </tr>
  <tr>
    <th>T1-d</th><th>T2-d</th><th>T1-d</th><th>T2-d</th><th>T1-d</th><th>T2-d</th><th>T1-d</th><th>T2-d</th><th>T1-d</th><th>T2-d</th><th>T1-d</th><th>T2-d</th><th>T1-d</th><th>T2-d</th><th>T1-d</th><th>T2-d</th><th>T1-d</th><th>T2-d</th><th>T1-d</th><th>T2-d</th><th>T1-d</th><th>T2-d</th><th>T1-d</th><th>T2-d</th><th>T1-d</th><th>T2-d</th><th>T1-d</th><th>T2-d</th>
  </tr>
  <tr>
    <td>Original</td>
    <td>4.03</td><td>3.58</td><td>8.29</td><td>4.59</td><td>6.06</td><td>3.32</td><td>13.13</td><td>9.05</td><td>14.60</td><td>13.00</td><td>11.19</td><td>7.08</td><td>23.91</td><td>8.08</td><td>11.32</td><td>6.57</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>7.24</td><td>2.91</td><td>14.48</td><td>3.54</td><td>6.73</td><td>3.16</td><td>20.66</td><td>14.18</td><td>21.55</td><td>15.83</td><td>24.31</td><td>15.06</td><td>25.65</td><td>6.92</td><td>12.03</td><td>6.49</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>5.73</td><td>3.11</td><td>12.41</td><td>1.98</td><td>6.27</td><td>2.40</td><td>19.28</td><td>13.30</td><td>23.44</td><td>18.10</td><td>22.84</td><td>16.67</td><td>23.41</td><td>8.61</td><td>12.51</td><td>5.22</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>5.30</td><td>2.36</td><td>8.54</td><td>3.33</td><td>9.97</td><td>5.68</td><td>16.92</td><td>12.25</td><td>21.64</td><td>17.00</td><td>24.07</td><td>15.08</td><td>22.65</td><td>9.50</td><td>12.34</td><td>6.85</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>5.58</td><td>2.65</td><td>5.81</td><td>4.38</td><td>5.68</td><td>1.56</td><td>15.02</td><td>8.84</td><td>20.07</td><td>13.97</td><td>21.34</td><td>14.82</td><td>26.01</td><td>9.60</td><td>11.62</td><td>6.61</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>6.72</td><td>3.67</td><td>4.54</td><td>1.42</td><td>7.95</td><td>4.31</td><td>23.45</td><td>9.75</td><td>20.90</td><td>17.54</td><td>23.59</td><td>16.28</td><td>21.90</td><td>8.13</td><td>13.06</td><td>8.62</td>
  </tr>
</table>

Table 3: WinoBias evaluation results (%). “T1” and “T2” denote the bias results on Type 1 and Type 2, “d” denotes the average difference between bias results of pro-stereotypical and anti-stereotypical.

variations, with maximal changes observed in OPT-2.7B (formal: -27.61%, informal: -23.87%). Sexual orientation bias follows closely, exhibiting significant fluctuations particularly under prefix modifications, with OPT-6.7B showing the largest increase (+14.28% with Prefix 2).

Religious bias demonstrates unique behavior under formal transformations. LLaMA-2-13B displays the most substantial reduction (-17.3%) in religious bias, contrasting with other bias types that typically respond more strongly to informal transformations. This distinctive pattern suggests that religious bias may be more deeply embedded in formal language structures, possibly reflecting how religious concepts are traditionally communicated in more formal contexts.

Nationality bias remains relatively stable, showing minimal changes across transformations (average variations <5%). This stability could indicate that nationality-related biases are more closely tied to semantic content rather than stylistic presentation, making them potentially more resistant to style-based mitigation strategies.

The MIN and MAX values in Table 2 highlight a clear hierarchy of style sensitivity among bias types. Gender and status biases consistently show the smallest transformational effects, suggesting these biases may be more deeply embedded in the fundamental semantic structure of the language models. In contrast, appearance, orientation, and religious biases regularly appear in the MAX category with changes often exceeding 20%, indicating higher susceptibility to stylistic manipulation. Age and disability biases occupy the middle ground, with average variations between 7-10% across all models and transformations.

7 Future Work

The findings of this study inspired several promising directions for future research. A critical next step involves developing new bias evaluation metrics that demonstrate robustness to stylistic variations while maintaining sensitivity to bias signals. These metrics should explicitly account for stylistic influence while providing reliable measurements across different contexts and applications.

Expanding the analysis to additional style dimensions beyond formality and structure would provide a more comprehensive understanding of style sensitivity in bias evaluation. Investigation of non-English languages and cross-lingual contexts would help establish the generalizability of our findings across different linguistic frameworks. Moreover, examining the interaction between style sensitivity and various model architecture choices could reveal important insights for model design.

The emergence of multimodal models presents another important avenue for future work. Investigating how style sensitivity manifests across different modalities and developing cross-modal bias evaluation frameworks that account for stylistic variations would extend the impact of this research. These future directions would significantly advance our understanding of bias evaluation in language models while addressing the current limitations of bias assessment methodologies.

8 Conclusion

Our comprehensive study on style sensitivity in bias evaluation metrics yields several critical insights that challenge current approaches to bias assessment in large language models. Current evaluation methods demonstrate significant sensitivity to stylistic variations even when semantic content remains unchanged, raising fundamental questions
about the reliability of existing bias evaluation metrics. Different style transformations exhibit varying impacts on bias scores, with informal style transformations consistently leading to substantial bias reductions across models, while formal transformations show complex, bias type dependent effects. Notably, even minimal changes like punctuation can significantly alter bias measurements.

The relationship between model size and bias sensitivity is particularly concerning. Larger models not only exhibit higher baseline biases but also show greater susceptibility to style transformations in bias scores. Moreover, certain bias categories, particularly appearance, sexual orientation, and religious bias, demonstrate heightened vulnerability to stylistic manipulations. These findings emphasize the urgent need for developing more robust bias evaluation metrics that can effectively decouple content from style.

Limitations

While our study provides valuable insights into the impact of style transformations on bias evaluation in NLP models, several limitations should be noted. Our analysis primarily focuses on a subset of commonly used language models and bias evaluation datasets, which may limit the generalizability of our findings to other models and datasets. The style transformation techniques employed represent only a fraction of the possible stylistic variations that could influence bias evaluations. Furthermore, the relationship between style and content may be more deeply intertwined than our current causal framework captures.

The causal inference techniques employed rely on specific assumptions that may not fully capture the complexity of biases in NLP models. While we conducted a rigorous human evaluation to ensure semantic preservation during style transformations, this process could introduce subjective biases. Additionally, our current tools for style transformation may introduce unintended artifacts that could affect our results.

Acknowledgments

We are deeply grateful to Zeyu Tang for his great help in this work. This work used the Dutch national e-infrastructure with the support of the SURF Cooperative using grant no. EINF-3953/L1.

References

Su Lin Blodgett, Gilsinla Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. Stereotyping norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1004–1015.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.

Prithviraj Damodaran. 2021. Parrot: Paraphrase generation for nlu.

Pieter Delobelle, Ewoenam Kwaku Tokpo, Toon Calders, and Bettina Berendt. 2022. Measuring fairness with biased rulers: A comparative study on bias metrics for pre-trained language models. In NAACL 2022: the 2022 Conference of the North American chapter of the Association for Computational Linguistics: human language technologies, pages 1693–1706.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

Robert F Engle, David F Hendry, and Jean-Francois Richard. 1983. Exogeneity. Econometrica: Journal of the Econometric Society, pages 277–304.

Isak Czeresnia Etinger and Alan W Black. 2019. Formality style transfer for noisy, user-generated conversations: Extracting labeled, parallel data from unlabeled corpora. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pages 11–16.

Seraphina Goldfarb-Tarrant, Rebecca Marchant, Ricardo Muñoz Sánchez, Mugdha Pandya, and Adam Lopez. 2020. Intrinsic bias metrics do not correlate with application bias. arXiv preprint arXiv:2012.15859.

Seraphina Goldfarb-Tarrant, Rebecca Marchant, Ricardo Muñoz Sánchez, Mugdha Pandya, and Adam Lopez. 2021. Intrinsic bias metrics do not correlate with application bias. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1926–1940.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213.

Bum Chul Kwon and Nandana Mihindukulasooriya. 2022. An empirical study on pseudo-log-likelihood bias measures for masked language models using paraphrased sentences. In Proceedings of the 2nd Workshop on Trustworthy Natural Language Processing (TrustNLP 2022), pages 74–79.

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pretrain, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35.

Chandler May, Alex Wang, Shikha Bordia, Samuel R Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. arXiv preprint arXiv:1903.10561.

Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. Stereotet: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.09456.

Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967, Online. Association for Computational Linguistics.

Judea Pearl. 2009. Causality. Cambridge University Press.

Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. 2017. Elements of Causal Inference: Foundations and Learning Algorithms. The MIT Press.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551.

Sudha Rao and Joel Tetreault. 2018. Dear sir or madam, may i introduce the gyafc dataset: Corpus, benchmarks and metrics for formality style transfer. arXiv preprint arXiv:1803.06535.

Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8–14, New Orleans, Louisiana. Association for Computational Linguistics.

Julian Salazar, Davis Liang, Toan Q Nguyen, and Katrin Kirchhoff. 2019. Masked language model scoring. arXiv preprint arXiv:1910.14659.

Julian Salazar, Davis Liang, Toan Q Nguyen, and Katrin Kirchhoff. 2020. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699–2712.

Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2023. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. arXiv preprint arXiv:2310.11324.

Preethi Seshadri, Pouya Pezeshkpour, and Sameer Singh. 2022. Quantifying social biases using templates is unreliable. In NeurIPS Workshop on Trustworthy and Socially Responsible Machine Learning (TSRML).

Zijing Shi, Meng Fang, Ling Chen, Yali Du, and Jun Wang. 2024. Human-guided moral decision making in text-based games. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 21574–21582.

Zijing Shi, Meng Fang, Yunqiu Xu, Ling Chen, and Yali Du. 2022. Stay moral and explore: Learn to behave morally in text-based games. In The Eleventh International Conference on Learning Representations.

Peter Spirtes, Clark Glymour, and Richard Scheines. 1993. Causation, Prediction, and Search. Springer New York.

Martina Toshevskaa and Sonja Gievska. 2021. A review of text style transfer using deep learning. IEEE Transactions on Artificial Intelligence, 3(5):669–684.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 33:12388–12401.

Alex Wang, Kyunghyun Cho, and CIFAR Azrieli Global Scholar. 2019. Bert has a mouth, and it must speak: Bert as a markov random field language model. NAACL HLT 2019, page 30.

Fei Wang, Wenjie Mo, Yiwei Wang, Wenxuan Zhou, and Muhao Chen. 2023. A causal view of entity bias in (large) language models. arXiv preprint arXiv:2305.14695.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher DeWan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mi-haylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre-trained transformer language models. Preprint, arXiv:2205.01068.

Jiaxu Zhao, Meng Fang, Shirui Pan, Wenpeng Yin, and Mykola Pechenizkiy. 2023a. Gptbias: A comprehensive framework for evaluating bias in large language models. arXiv preprint arXiv:2312.06315.

Jiaxu Zhao, Meng Fang, Zijing Shi, Yitong Li, Ling Chen, and Mykola Pechenizkiy. 2023b. CHBias: Bias evaluation and mitigation of Chinese conversational language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

Jiaxu Zhao, Zijing Shi, Yitong Li, Yulong Pei, Ling Chen, Meng Fang, and Mykola Pechenizkiy. 2024. More than minorities and majorities: Understanding multilateral bias in language generation. In Findings of the Association for Computational Linguistics ACL 2024, pages 9987–10001.

Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018a. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 15–20.

Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018b. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 15–20, New Orleans, Louisiana. Association for Computational Linguistics.

Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning, pages 12697–12706. PMLR.
A  Case Study of Style Transformations

Table 4 shows a case study of style transformations, with the original sentence of “Stereotype: The man was highly skilled in CAD engineering. He was considered one of the best engineers in his field. Anti-stereotype: The woman was highly skilled in CAD engineering. She was considered one of the best engineers in her field.”

B  Human Annotation Guidelines for Formality Transformation

This section provides guidelines for human annotators to evaluate and validate formality transformations of sentences. The goal is to ensure that transformed sentences maintain the same semantic meaning as the original while appropriately changing formality levels. We have 3 annotators to annotate both formal-original and informal-original sentence pairs in CrowS-Pairs. Annotation was conducted by three PhD students in the Computer Science department at our university. The students can speak fluent English with training in semantic analysis and annotation. They agreed to list them in the acknowledgment after publishing this work. No additional monetary compensation was provided.

B.1  Annotation Instructions

Step 1 (Semantic Consistency Annotation)  Annotators must categorize each sentence transformation based on semantic preservation using the following labels:

Label 0 (Same): The generated sentence accurately preserves the meaning of the original sentence.

Label 1 (Different): The generated sentence alters the original meaning.

Label 2 (Not Sure): The meaning of the transformed sentence is ambiguous or unclear.

For each pair of original and transformed sentences, annotators should carefully assess whether key details, intent, and sentiment remain unchanged.

Step 2 (Rewriting)  If Label 0 (Same) is assigned: No further modification is required; retain the sentence as is.

If Label 1 (Different) or Label 2 (Not Sure) is assigned: Annotators should rewrite the transformed sentence to ensure it accurately preserves the meaning of the original sentence while maintaining the intended formality.

B.2  Guidelines for Rewriting

• Ensure that the revised sentence retains the same meaning as the original.

• Adjust only the formality level without introducing new information or omitting critical details.

• Maintain naturalness and fluency in the rewritten sentences.

• Preserve the grammatical correctness of the sentence.

B.3  Quality Control

Our results (Table 5) demonstrate strong annotation consistency across both formal and informal style transformations. For formal style transformations, annotators marked 92.5% of the pairs as semantically equivalent (Label 0), with only 6.2% identified as having different meanings (Label 1) and 1.3% marked as ambiguous (Label 2). The informal style transformations showed even higher agreement, with 95.3% of pairs labeled as semantically equivalent, 3.4% as different, and 1.3% as ambiguous.

To ensure data quality, we implemented a two-stage annotation process: sentence pairs unanimously labeled as Same were retained for our analysis, while all other cases (those receiving Different or Not Sure labels, or having disagreement among annotators) underwent a rewriting process by the three annotators to preserve the intended style transformation while maintaining semantic equivalence with the original text.

C  T-values of Student’s Two-tailed Test

We show the t-value of Student’s two-tail test for various styles and content of CrowS-Pairs in Tables 6 and 7.

D  CrowS-Pairs Evaluation Results

The detailed CrowS-Pairs evaluation results are shown in Tables 8 and 9.

E  Results of WinoBias

In this section, we report the detailed results of WinoBias. Table 10 presents the results of GPT-2; Table 11 for the results of LLaMA-2-7B; Table 12 for the results of LLaMA-2-13B; Table 13 for the results of OPT-2.7B; Table 14 for the results of OPT-6.7B; Table 15 for the results of OPT-13B;
<table>
  <tr>
    <th>Style</th>
    <th>Stereotype</th>
    <th>Anti-stereotype</th>
  </tr>
  <tr>
    <td>Original</td>
    <td>The <span style="background-color: yellow">man</span> was highly skilled in CAD engineering. <span style="background-color: green">He</span> was considered one of the best engineers in <span style="background-color: yellow">his</span> field.</td>
    <td>The <span style="background-color: green">woman</span> was highly skilled in CAD engineering. <span style="background-color: yellow">She</span> was considered one of the best engineers in <span style="background-color: green">her</span> field.</td>
  </tr>
  <tr>
    <td>Informal</td>
    <td>The man was <span style="background-color: yellow">really good at</span> CAD engineering. <span style="background-color: green">People</span> thought he was one of the top engineers in his field.</td>
    <td>The woman was <span style="background-color: yellow">really good at</span> CAD engineering. <span style="background-color: green">People</span> thought she was one of the top engineers in her field.</td>
  </tr>
  <tr>
    <td>Formal</td>
    <td>The man <span style="background-color: yellow">possessed exceptional proficiency</span> in CAD engineering and was regarded as one of the <span style="background-color: yellow">foremost</span> engineers in his field.</td>
    <td>The woman <span style="background-color: yellow">possessed exceptional proficiency</span> in CAD engineering and was regarded as one of the <span style="background-color: yellow">foremost</span> engineers in her field.</td>
  </tr>
  <tr>
    <td>Prefix 1</td>
    <td><span style="background-color: yellow">It's important to acknowledge that:</span> The man was highly skilled in CAD engineering. He was considered one of the best engineers in his field.</td>
    <td><span style="background-color: yellow">It's important to acknowledge that:</span> The woman was highly skilled in CAD engineering. She was considered one of the best engineers in her field.</td>
  </tr>
  <tr>
    <td>Prefix 2</td>
    <td><span style="background-color: yellow">It's worth noting that:</span> The man was highly skilled in CAD engineering. He was considered one of the best engineers in his field.</td>
    <td><span style="background-color: yellow">It's worth noting that:</span> The woman was highly skilled in CAD engineering. She was considered one of the best engineers in her field.</td>
  </tr>
  <tr>
    <td>Prefix 3</td>
    <td><span style="background-color: yellow">With that in mind:</span> The man was highly skilled in CAD engineering. He was considered one of the best engineers in his field.</td>
    <td><span style="background-color: yellow">With that in mind:</span> The woman was highly skilled in CAD engineering. She was considered one of the best engineers in her field.</td>
  </tr>
  <tr>
    <td>Punctuation</td>
    <td>The man was highly skilled in CAD engineering. He was considered one of the best engineers in his field <span style="background-color: yellow">!</span></td>
    <td>The woman was highly skilled in CAD engineering. She was considered one of the best engineers in her field <span style="background-color: yellow">!</span></td>
  </tr>
  <tr>
    <td>Evident</td>
    <td><span style="background-color: yellow">Apparently,</span> The man was highly skilled in CAD engineering. He was considered one of the best engineers in his field.</td>
    <td><span style="background-color: yellow">Apparently</span> The woman was highly skilled in CAD engineering. She was considered one of the best engineers in her field.</td>
  </tr>
  <tr>
    <td>QA</td>
    <td><span style="background-color: yellow">Want to hear something?</span> The man was highly skilled in CAD engineering. He was considered one of the best engineers in his field.</td>
    <td><span style="background-color: yellow">Want to hear something?</span> The woman was highly skilled in CAD engineering. She was considered one of the best engineers in her field.</td>
  </tr>
</table>

Table 4: Case study of style transformations. “Stereotype” represents the stereotyped sample in Crows-Pairs and the corresponding content after the style transfer, and conversely, the “Anti-stereotype” is the anti-stereotyped sample. Highlighted text in <span style="background-color: green">green</span> is a word or phrase that is different between the stereotypical sentence and the anti-stereotypical sentence in Crows-Pairs, which is usually a word that represents a demographic group. <span style="background-color: yellow">Yellow</span> highlighted text is words and phrases that have been altered by stylistic transformation.

<table>
  <tr>
    <th rowspan="2">Label</th>
    <th colspan="4">Formal</th>
    <th colspan="4">Informal</th>
  </tr>
  <tr>
    <th>A1</th>
    <th>A2</th>
    <th>A3</th>
    <th>Avg(%)</th>
    <th>A1</th>
    <th>A2</th>
    <th>A3</th>
    <th>Avg(%)</th>
  </tr>
  <tr>
    <td>Same (0)</td>
    <td>1,376</td>
    <td>1,415</td>
    <td>1,392</td>
    <td>92.5</td>
    <td>1,433</td>
    <td>1,440</td>
    <td>1,437</td>
    <td>95.3</td>
  </tr>
  <tr>
    <td>Different (1)</td>
    <td>107</td>
    <td>82</td>
    <td>93</td>
    <td>6.2</td>
    <td>59</td>
    <td>37</td>
    <td>60</td>
    <td>3.4</td>
  </tr>
  <tr>
    <td>Not Sure (2)</td>
    <td>25</td>
    <td>11</td>
    <td>23</td>
    <td>1.3</td>
    <td>16</td>
    <td>31</td>
    <td>11</td>
    <td>1.3</td>
  </tr>
</table>

Table 5: Distribution of human annotations for formality transfer.

Table 17 for the results of LLaMA-3.1-8B; and Table 16 for the results of Mistral-7B;

F PLL results

Some results of PLL are visualized in Figure 3 and 4.
<table>
  <tr>
    <th>Style</th>
    <th>Bias</th>
    <th>GPT-2</th>
    <th>L-2-TB</th>
    <th>L-2-13B</th>
    <th>OPT-2.7B</th>
    <th>OPT-6.7B</th>
    <th>OPT-13B</th>
    <th>L-3.1-8B</th>
    <th>M-7B</th>
  </tr>
  <tr>
    <td rowspan="9">original stereo/original anti</td>
    <td>gender</td>
    <td>-0.21</td>
    <td>0.05</td>
    <td>1.33</td>
    <td>0.28</td>
    <td>0.43</td>
    <td>0.46</td>
    <td>0.35</td>
    <td>0.38</td>
  </tr>
  <tr>
    <td>race</td>
    <td>1.23</td>
    <td>2.30</td>
    <td>1.79</td>
    <td>1.88</td>
    <td>1.25</td>
    <td>2.20</td>
    <td>1.85</td>
    <td>2.11</td>
  </tr>
  <tr>
    <td>religion</td>
    <td>1.02</td>
    <td>1.45</td>
    <td>2.15</td>
    <td>1.28</td>
    <td>1.64</td>
    <td>2.39</td>
    <td>1.40</td>
    <td>1.75</td>
  </tr>
  <tr>
    <td>status</td>
    <td>0.60</td>
    <td>1.75</td>
    <td>2.09</td>
    <td>1.22</td>
    <td>1.78</td>
    <td>1.83</td>
    <td>0.98</td>
    <td>1.31</td>
  </tr>
  <tr>
    <td>disability</td>
    <td>0.74</td>
    <td>2.30</td>
    <td>2.89</td>
    <td>2.40</td>
    <td>2.38</td>
    <td>2.86</td>
    <td>2.05</td>
    <td>2.20</td>
  </tr>
  <tr>
    <td>nationality</td>
    <td>-0.01</td>
    <td>1.75</td>
    <td>1.55</td>
    <td>0.69</td>
    <td>0.66</td>
    <td>0.73</td>
    <td>1.38</td>
    <td>1.38</td>
  </tr>
  <tr>
    <td>orientation</td>
    <td>1.05</td>
    <td>0.86</td>
    <td>1.35</td>
    <td>0.94</td>
    <td>0.51</td>
    <td>0.63</td>
    <td>0.61</td>
    <td>0.52</td>
  </tr>
  <tr>
    <td>appearance</td>
    <td>0.74</td>
    <td>1.00</td>
    <td>2.05</td>
    <td>1.48</td>
    <td>1.25</td>
    <td>2.10</td>
    <td>0.40</td>
    <td>1.40</td>
  </tr>
  <tr>
    <td>age</td>
    <td>0.47</td>
    <td>1.76</td>
    <td>2.68</td>
    <td>0.80</td>
    <td>1.13</td>
    <td>0.63</td>
    <td>0.32</td>
    <td>1.26</td>
  </tr>
  <tr>
    <td rowspan="9">informal stereo/formal stereo</td>
    <td>gender</td>
    <td>1.18</td>
    <td>-0.42</td>
    <td>-0.45</td>
    <td>0.35</td>
    <td>0.24</td>
    <td>0.45</td>
    <td>-0.70</td>
    <td>-0.49</td>
  </tr>
  <tr>
    <td>race</td>
    <td>7.22</td>
    <td>2.33</td>
    <td>2.33</td>
    <td>5.06</td>
    <td>5.42</td>
    <td>5.55</td>
    <td>3.32</td>
    <td>3.42</td>
  </tr>
  <tr>
    <td>religion</td>
    <td>3.60</td>
    <td>1.35</td>
    <td>1.40</td>
    <td>1.63</td>
    <td>1.69</td>
    <td>1.73</td>
    <td>1.43</td>
    <td>1.56</td>
  </tr>
  <tr>
    <td>status</td>
    <td>3.84</td>
    <td>2.13</td>
    <td>1.62</td>
    <td>3.76</td>
    <td>3.08</td>
    <td>2.98</td>
    <td>1.67</td>
    <td>1.68</td>
  </tr>
  <tr>
    <td>disability</td>
    <td>4.14</td>
    <td>1.60</td>
    <td>1.41</td>
    <td>2.45</td>
    <td>2.73</td>
    <td>2.66</td>
    <td>1.18</td>
    <td>1.38</td>
  </tr>
  <tr>
    <td>nationality</td>
    <td>5.30</td>
    <td>3.48</td>
    <td>4.11</td>
    <td>3.35</td>
    <td>3.98</td>
    <td>4.35</td>
    <td>3.16</td>
    <td>3.61</td>
  </tr>
  <tr>
    <td>orientation</td>
    <td>2.61</td>
    <td>2.37</td>
    <td>2.46</td>
    <td>1.49</td>
    <td>2.03</td>
    <td>2.17</td>
    <td>2.33</td>
    <td>2.70</td>
  </tr>
  <tr>
    <td>appearance</td>
    <td>0.94</td>
    <td>0.64</td>
    <td>0.25</td>
    <td>1.49</td>
    <td>1.17</td>
    <td>1.18</td>
    <td>0.52</td>
    <td>0.66</td>
  </tr>
  <tr>
    <td>age</td>
    <td>2.38</td>
    <td>1.80</td>
    <td>1.53</td>
    <td>0.32</td>
    <td>0.88</td>
    <td>0.95</td>
    <td>0.58</td>
    <td>0.88</td>
  </tr>
  <tr>
    <td rowspan="9">informal anti/formal anti</td>
    <td>gender</td>
    <td>1.03</td>
    <td>0.19</td>
    <td>-0.48</td>
    <td>0.32</td>
    <td>0.20</td>
    <td>0.40</td>
    <td>-0.03</td>
    <td>0.17</td>
  </tr>
  <tr>
    <td>race</td>
    <td>8.45</td>
    <td>4.11</td>
    <td>4.03</td>
    <td>5.14</td>
    <td>5.01</td>
    <td>5.04</td>
    <td>3.96</td>
    <td>4.15</td>
  </tr>
  <tr>
    <td>religion</td>
    <td>3.78</td>
    <td>1.53</td>
    <td>0.52</td>
    <td>2.16</td>
    <td>1.30</td>
    <td>1.32</td>
    <td>1.12</td>
    <td>1.53</td>
  </tr>
  <tr>
    <td>status</td>
    <td>4.16</td>
    <td>2.83</td>
    <td>2.73</td>
    <td>3.02</td>
    <td>4.09</td>
    <td>4.25</td>
    <td>2.39</td>
    <td>2.35</td>
  </tr>
  <tr>
    <td>disability</td>
    <td>2.92</td>
    <td>1.23</td>
    <td>1.04</td>
    <td>2.30</td>
    <td>1.68</td>
    <td>1.84</td>
    <td>1.32</td>
    <td>1.53</td>
  </tr>
  <tr>
    <td>nationality</td>
    <td>4.66</td>
    <td>2.18</td>
    <td>2.03</td>
    <td>1.95</td>
    <td>3.06</td>
    <td>3.19</td>
    <td>1.88</td>
    <td>2.35</td>
  </tr>
  <tr>
    <td>orientation</td>
    <td>2.57</td>
    <td>2.96</td>
    <td>2.93</td>
    <td>2.66</td>
    <td>2.66</td>
    <td>2.57</td>
    <td>2.65</td>
    <td>2.74</td>
  </tr>
  <tr>
    <td>appearance</td>
    <td>2.04</td>
    <td>1.60</td>
    <td>0.33</td>
    <td>1.06</td>
    <td>1.92</td>
    <td>2.30</td>
    <td>1.41</td>
    <td>1.56</td>
  </tr>
  <tr>
    <td>age</td>
    <td>5.39</td>
    <td>3.04</td>
    <td>0.40</td>
    <td>1.16</td>
    <td>1.56</td>
    <td>1.73</td>
    <td>0.00</td>
    <td>0.12</td>
  </tr>
  <tr>
    <td rowspan="9">original stereo/informal stereo</td>
    <td>gender</td>
    <td>0.11</td>
    <td>-0.62</td>
    <td>-0.21</td>
    <td>0.32</td>
    <td>0.88</td>
    <td>0.25</td>
    <td>-0.33</td>
    <td>-0.53</td>
  </tr>
  <tr>
    <td>race</td>
    <td>7.03</td>
    <td>5.47</td>
    <td>4.54</td>
    <td>5.89</td>
    <td>6.31</td>
    <td>6.64</td>
    <td>5.29</td>
    <td>5.53</td>
  </tr>
  <tr>
    <td>religion</td>
    <td>3.40</td>
    <td>2.45</td>
    <td>2.37</td>
    <td>2.27</td>
    <td>1.85</td>
    <td>2.50</td>
    <td>2.05</td>
    <td>2.10</td>
  </tr>
  <tr>
    <td>status</td>
    <td>2.72</td>
    <td>3.34</td>
    <td>2.58</td>
    <td>3.02</td>
    <td>3.19</td>
    <td>3.67</td>
    <td>2.58</td>
    <td>3.58</td>
  </tr>
  <tr>
    <td>disability</td>
    <td>2.53</td>
    <td>1.19</td>
    <td>1.41</td>
    <td>1.74</td>
    <td>2.65</td>
    <td>2.58</td>
    <td>1.16</td>
    <td>1.38</td>
  </tr>
  <tr>
    <td>nationality</td>
    <td>5.25</td>
    <td>4.85</td>
    <td>3.99</td>
    <td>4.74</td>
    <td>3.62</td>
    <td>4.61</td>
    <td>4.82</td>
    <td>5.10</td>
  </tr>
  <tr>
    <td>orientation</td>
    <td>2.43</td>
    <td>1.91</td>
    <td>2.42</td>
    <td>1.61</td>
    <td>2.65</td>
    <td>3.39</td>
    <td>1.48</td>
    <td>1.86</td>
  </tr>
  <tr>
    <td>appearance</td>
    <td>3.19</td>
    <td>1.59</td>
    <td>1.26</td>
    <td>1.33</td>
    <td>1.37</td>
    <td>1.68</td>
    <td>1.02</td>
    <td>1.80</td>
  </tr>
  <tr>
    <td>age</td>
    <td>1.47</td>
    <td>2.04</td>
    <td>1.09</td>
    <td>1.83</td>
    <td>1.27</td>
    <td>1.34</td>
    <td>1.90</td>
    <td>1.85</td>
  </tr>
  <tr>
    <td rowspan="9">original stereo/informal anti</td>
    <td>gender</td>
    <td>-0.13</td>
    <td>0.06</td>
    <td>1.29</td>
    <td>0.35</td>
    <td>0.52</td>
    <td>0.59</td>
    <td>-0.23</td>
    <td>-0.28</td>
  </tr>
  <tr>
    <td>race</td>
    <td>7.74</td>
    <td>5.78</td>
    <td>5.91</td>
    <td>7.63</td>
    <td>8.59</td>
    <td>8.08</td>
    <td>5.37</td>
    <td>4.56</td>
  </tr>
  <tr>
    <td>religion</td>
    <td>3.78</td>
    <td>2.34</td>
    <td>3.33</td>
    <td>3.49</td>
    <td>2.58</td>
    <td>3.88</td>
    <td>2.13</td>
    <td>2.43</td>
  </tr>
  <tr>
    <td>status</td>
    <td>4.24</td>
    <td>3.92</td>
    <td>4.61</td>
    <td>4.45</td>
    <td>5.45</td>
    <td>5.44</td>
    <td>4.67</td>
    <td>3.88</td>
  </tr>
  <tr>
    <td>disability</td>
    <td>2.19</td>
    <td>3.57</td>
    <td>3.77</td>
    <td>4.06</td>
    <td>4.11</td>
    <td>5.35</td>
    <td>2.67</td>
    <td>4.07</td>
  </tr>
  <tr>
    <td>nationality</td>
    <td>4.95</td>
    <td>4.28</td>
    <td>3.48</td>
    <td>3.69</td>
    <td>2.80</td>
    <td>4.30</td>
    <td>4.09</td>
    <td>4.00</td>
  </tr>
  <tr>
    <td>orientation</td>
    <td>4.54</td>
    <td>3.70</td>
    <td>4.43</td>
    <td>5.08</td>
    <td>4.99</td>
    <td>3.50</td>
    <td>3.59</td>
    <td>3.59</td>
  </tr>
  <tr>
    <td>appearance</td>
    <td>2.86</td>
    <td>2.42</td>
    <td>2.40</td>
    <td>2.46</td>
    <td>3.11</td>
    <td>3.45</td>
    <td>1.26</td>
    <td>2.25</td>
  </tr>
  <tr>
    <td>age</td>
    <td>2.33</td>
    <td>1.95</td>
    <td>2.66</td>
    <td>2.05</td>
    <td>2.14</td>
    <td>4.14</td>
    <td>3.48</td>
    <td>1.46</td>
  </tr>
  <tr>
    <td rowspan="9">original anti/informal anti</td>
    <td>gender</td>
    <td>0.88</td>
    <td>-0.31</td>
    <td>0.12</td>
    <td>0.50</td>
    <td>0.78</td>
    <td>0.85</td>
    <td>-0.59</td>
    <td>-0.57</td>
  </tr>
  <tr>
    <td>race</td>
    <td>6.40</td>
    <td>4.89</td>
    <td>3.97</td>
    <td>5.80</td>
    <td>6.18</td>
    <td>5.22</td>
    <td>3.61</td>
    <td>4.68</td>
  </tr>
  <tr>
    <td>religion</td>
    <td>3.42</td>
    <td>1.00</td>
    <td>0.71</td>
    <td>2.21</td>
    <td>2.04</td>
    <td>2.89</td>
    <td>0.30</td>
    <td>0.87</td>
  </tr>
  <tr>
    <td>status</td>
    <td>3.24</td>
    <td>3.22</td>
    <td>1.84</td>
    <td>3.02</td>
    <td>4.06</td>
    <td>4.46</td>
    <td>2.99</td>
    <td>3.41</td>
  </tr>
  <tr>
    <td>disability</td>
    <td>1.20</td>
    <td>0.94</td>
    <td>1.61</td>
    <td>2.15</td>
    <td>2.43</td>
    <td>2.24</td>
    <td>0.65</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>nationality</td>
    <td>7.47</td>
    <td>2.57</td>
    <td>3.09</td>
    <td>2.09</td>
    <td>3.04</td>
    <td>2.08</td>
    <td>2.01</td>
    <td>3.01</td>
  </tr>
  <tr>
    <td>orientation</td>
    <td>2.84</td>
    <td>2.85</td>
    <td>2.59</td>
    <td>4.01</td>
    <td>4.12</td>
    <td>4.40</td>
    <td>1.66</td>
    <td>3.13</td>
  </tr>
  <tr>
    <td>appearance</td>
    <td>2.12</td>
    <td>0.82</td>
    <td>0.62</td>
    <td>2.12</td>
    <td>1.50</td>
    <td>2.86</td>
    <td>1.40</td>
    <td>0.77</td>
  </tr>
  <tr>
    <td>age</td>
    <td>2.81</td>
    <td>1.59</td>
    <td>0.73</td>
    <td>2.48</td>
    <td>1.89</td>
    <td>1.96</td>
    <td>1.32</td>
    <td>1.29</td>
  </tr>
  <tr>
    <td rowspan="9">original stereo/formal stereo</td>
    <td>gender</td>
    <td>0.14</td>
    <td>-0.15</td>
    <td>0.44</td>
    <td>0.02</td>
    <td>-0.11</td>
    <td>0.22</td>
    <td>-0.46</td>
    <td>-0.34</td>
  </tr>
  <tr>
    <td>race</td>
    <td>-0.32</td>
    <td>2.75</td>
    <td>2.82</td>
    <td>1.13</td>
    <td>0.91</td>
    <td>0.96</td>
    <td>2.41</td>
    <td>2.47</td>
  </tr>
  <tr>
    <td>religion</td>
    <td>-0.48</td>
    <td>1.71</td>
    <td>0.82</td>
    <td>1.56</td>
    <td>0.69</td>
    <td>0.86</td>
    <td>1.69</td>
    <td>1.76</td>
  </tr>
  <tr>
    <td>status</td>
    <td>-0.66</td>
    <td>1.36</td>
    <td>0.62</td>
    <td>0.07</td>
    <td>0.72</td>
    <td>1.07</td>
    <td>0.88</td>
    <td>0.78</td>
  </tr>
  <tr>
    <td>disability</td>
    <td>-2.91</td>
    <td>-0.17</td>
    <td>-0.25</td>
    <td>0.11</td>
    <td>0.38</td>
    <td>0.42</td>
    <td>-0.20</td>
    <td>-0.22</td>
  </tr>
  <tr>
    <td>nationality</td>
    <td>-0.33</td>
    <td>0.87</td>
    <td>0.53</td>
    <td>0.07</td>
    <td>0.82</td>
    <td>1.30</td>
    <td>0.45</td>
    <td>0.86</td>
  </tr>
  <tr>
    <td>orientation</td>
    <td>-1.01</td>
    <td>-0.59</td>
    <td>0.07</td>
    <td>0.68</td>
    <td>0.59</td>
    <td>0.84</td>
    <td>-0.62</td>
    <td>-0.34</td>
  </tr>
  <tr>
    <td>appearance</td>
    <td>-0.09</td>
    <td>1.01</td>
    <td>0.49</td>
    <td>0.49</td>
    <td>0.76</td>
    <td>0.92</td>
    <td>0.66</td>
    <td>0.97</td>
  </tr>
  <tr>
    <td>age</td>
    <td>0.12</td>
    <td>-0.14</td>
    <td>0.24</td>
    <td>0.03</td>
    <td>0.16</td>
    <td>0.64</td>
    <td>-0.60</td>
    <td>-0.23</td>
  </tr>
  <tr>
    <td rowspan="9">original anti/formal anti</td>
    <td>gender</td>
    <td>0.04</td>
    <td>0.56</td>
    <td>-0.03</td>
    <td>0.26</td>
    <td>0.23</td>
    <td>0.42</td>
    <td>0.65</td>
    <td>1.03</td>
  </tr>
  <tr>
    <td>race</td>
    <td>-1.12</td>
    <td>0.89</td>
    <td>1.03</td>
    <td>0.69</td>
    <td>0.33</td>
    <td>0.51</td>
    <td>0.80</td>
    <td>0.86</td>
  </tr>
  <tr>
    <td>religion</td>
    <td>-0.85</td>
    <td>1.12</td>
    <td>0.24</td>
    <td>0.97</td>
    <td>0.14</td>
    <td>0.52</td>
    <td>1.16</td>
    <td>1.54</td>
  </tr>
  <tr>
    <td>status</td>
    <td>-0.91</td>
    <td>0.94</td>
    <td>0.07</td>
    <td>0.44</td>
    <td>0.13</td>
    <td>0.49</td>
    <td>0.55</td>
    <td>0.96</td>
  </tr>
  <tr>
    <td>disability</td>
    <td>-0.59</td>
    <td>0.01</td>
    <td>0.79</td>
    <td>0.03</td>
    <td>0.16</td>
    <td>0.60</td>
    <td>-0.48</td>
    <td>-0.49</td>
  </tr>
  <tr>
    <td>nationality</td>
    <td>-0.30</td>
    <td>1.16</td>
    <td>0.77</td>
    <td>1.18</td>
    <td>1.27</td>
    <td>1.24</td>
    <td>0.84</td>
    <td>1.17</td>
  </tr>
  <tr>
    <td>orientation</td>
    <td>0.22</td>
    <td>0.03</td>
    <td>0.41</td>
    <td>-0.01</td>
    <td>-0.01</td>
    <td>0.07</td>
    <td>-0.24</td>
    <td>0.07</td>
  </tr>
  <tr>
    <td>appearance</td>
    <td>0.27</td>
    <td>0.54</td>
    <td>0.86</td>
    <td>0.00</td>
    <td>0.48</td>
    <td>0.85</td>
    <td>0.16</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td>age</td>
    <td>-0.05</td>
    <td>1.31</td>
    <td>1.19</td>
    <td>0.67</td>
    <td>0.76</td>
    <td>0.86</td>
    <td>1.18</td>
    <td>1.32</td>
  </tr>
</table>

Table 6: T-value of student’s two-tail test (1).
<table>
  <tr>
    <th rowspan="2">Style</th>
    <th rowspan="2">Bias</th>
    <th colspan="7">GPT-2</th>
    <th colspan="7">L-2-7B</th>
    <th colspan="7">L-2-13B</th>
    <th colspan="7">OPF2.7B</th>
    <th colspan="7">OPF6.7B</th>
    <th colspan="7">OPF13B</th>
    <th colspan="7">L-3-13B</th>
    <th colspan="7">M-7B</th>
  </tr>
  <tr>
    <th>GPT-2</th><th>L-2-7B</th><th>L-2-13B</th><th>OPF2.7B</th><th>OPF6.7B</th><th>OPF13B</th><th>L-3-13B</th><th>M-7B</th>
  </tr>
  <tr>
    <td rowspan="8">original stereo/formal anti</td>
    <td>gender</td><td>0.35</td><td>1.17</td><td>0.69</td><td>0.46</td><td>0.03</td><td>0.84</td><td>0.78</td><td>0.95</td>
  </tr>
  <tr>
    <td>race</td><td>0.13</td><td>2.81</td><td>3.60</td><td>-2.47</td><td>3.43</td><td>3.66</td><td>2.36</td><td>2.75</td>
  </tr>
  <tr>
    <td>religion</td><td>0.33</td><td>2.48</td><td>3.39</td><td>1.13</td><td>1.03</td><td>1.22</td><td>2.05</td><td>2.35</td>
  </tr>
  <tr>
    <td>status</td><td>0.05</td><td>2.85</td><td>2.74</td><td>1.11</td><td>2.27</td><td>2.68</td><td>2.41</td><td>2.90</td>
  </tr>
  <tr>
    <td>disability</td><td>-0.02</td><td>2.59</td><td>3.12</td><td>1.59</td><td>2.18</td><td>2.24</td><td>2.25</td><td>2.59</td>
  </tr>
  <tr>
    <td>nationality</td><td>-0.32</td><td>2.54</td><td>2.73</td><td>1.15</td><td>1.28</td><td>1.59</td><td>2.42</td><td>2.81</td>
  </tr>
  <tr>
    <td>orientation</td><td>1.08</td><td>1.41</td><td>0.79</td><td>0.99</td><td>2.04</td><td>2.23</td><td>1.49</td><td>1.43</td>
  </tr>
  <tr>
    <td>appearance</td><td>0.22</td><td>1.24</td><td>1.71</td><td>1.59</td><td>1.34</td><td>1.31</td><td>0.85</td><td>0.91</td>
  </tr>
  <tr>
    <td>age</td><td>-0.22</td><td>1.81</td><td>2.98</td><td>2.08</td><td>1.84</td><td>1.83</td><td>1.47</td><td>1.50</td>
  </tr>
  <tr>
    <td rowspan="8">original anti/formal stereo</td>
    <td>gender</td><td>-0.45</td><td>-0.07</td><td>-0.55</td><td>-0.64</td><td>-0.21</td><td>0.22</td><td>-0.03</td><td>0.21</td>
  </tr>
  <tr>
    <td>race</td><td>-1.15</td><td>0.27</td><td>0.27</td><td>-0.24</td><td>-1.21</td><td>-1.16</td><td>0.15</td><td>0.55</td>
  </tr>
  <tr>
    <td>religion</td><td>-0.26</td><td>-0.86</td><td>-1.46</td><td>-0.09</td><td>-0.40</td><td>-0.02</td><td>-1.22</td><td>-1.18</td>
  </tr>
  <tr>
    <td>status</td><td>-0.91</td><td>-1.43</td><td>-1.27</td><td>-1.40</td><td>-1.46</td><td>-1.38</td><td>-1.82</td><td>-1.49</td>
  </tr>
  <tr>
    <td>disability</td><td>-3.30</td><td>-1.91</td><td>-3.48</td><td>-2.32</td><td>-2.43</td><td>-2.44</td><td>-2.26</td><td>-2.24</td>
  </tr>
  <tr>
    <td>nationality</td><td>-0.58</td><td>-0.70</td><td>-1.13</td><td>-0.27</td><td>-0.52</td><td>-0.58</td><td>-1.19</td><td>-0.78</td>
  </tr>
  <tr>
    <td>orientation</td><td>-2.25</td><td>-1.07</td><td>-1.66</td><td>-0.05</td><td>-0.65</td><td>-0.45</td><td>-1.46</td><td>-0.98</td>
  </tr>
  <tr>
    <td>appearance</td><td>-0.41</td><td>-0.91</td><td>-0.93</td><td>-0.28</td><td>-0.08</td><td>-0.10</td><td>-0.46</td><td>-0.42</td>
  </tr>
  <tr>
    <td>age</td><td>0.29</td><td>-0.38</td><td>-1.85</td><td>-0.63</td><td>-0.88</td><td>-0.40</td><td>-0.39</td><td>-0.48</td>
  </tr>
  <tr>
    <td rowspan="8">original anti/informal stereo</td>
    <td>gender</td><td>1.14</td><td>-0.80</td><td>-0.95</td><td>-0.40</td><td>-0.51</td><td>-0.29</td><td>-0.85</td><td>-0.91</td>
  </tr>
  <tr>
    <td>race</td><td>6.45</td><td>3.39</td><td>3.20</td><td>4.28</td><td>4.30</td><td>4.77</td><td>3.10</td><td>3.13</td>
  </tr>
  <tr>
    <td>religion</td><td>2.42</td><td>0.46</td><td>-0.35</td><td>1.62</td><td>0.87</td><td>1.07</td><td>0.25</td><td>0.75</td>
  </tr>
  <tr>
    <td>status</td><td>2.00</td><td>0.88</td><td>0.42</td><td>2.19</td><td>1.53</td><td>1.44</td><td>0.38</td><td>0.77</td>
  </tr>
  <tr>
    <td>disability</td><td>1.54</td><td>-1.08</td><td>-1.90</td><td>-0.40</td><td>0.20</td><td>0.66</td><td>-1.09</td><td>-0.73</td>
  </tr>
  <tr>
    <td>nationality</td><td>4.67</td><td>3.44</td><td>3.04</td><td>3.26</td><td>3.44</td><td>3.89</td><td>3.45</td><td>3.49</td>
  </tr>
  <tr>
    <td>orientation</td><td>1.41</td><td>0.79</td><td>1.13</td><td>0.96</td><td>1.54</td><td>1.76</td><td>0.63</td><td>0.63</td>
  </tr>
  <tr>
    <td>appearance</td><td>0.53</td><td>0.52</td><td>-0.46</td><td>1.06</td><td>0.59</td><td>1.00</td><td>0.59</td><td>0.67</td>
  </tr>
  <tr>
    <td>age</td><td>2.04</td><td>0.83</td><td>-0.96</td><td>0.57</td><td>0.34</td><td>0.33</td><td>0.59</td><td>0.88</td>
  </tr>
  <tr>
    <td rowspan="8">informal stereo/informal anti</td>
    <td>gender</td><td>0.35</td><td>0.38</td><td>0.32</td><td>1.13</td><td>0.17</td><td>0.57</td><td>0.43</td><td>0.63</td>
  </tr>
  <tr>
    <td>race</td><td>0.34</td><td>0.63</td><td>2.31</td><td>1.59</td><td>1.93</td><td>2.01</td><td>0.17</td><td>0.65</td>
  </tr>
  <tr>
    <td>religion</td><td>1.04</td><td>1.05</td><td>1.71</td><td>0.82</td><td>1.38</td><td>1.88</td><td>0.67</td><td>1.06</td>
  </tr>
  <tr>
    <td>status</td><td>1.76</td><td>2.45</td><td>2.26</td><td>1.81</td><td>1.74</td><td>1.86</td><td>2.16</td><td>2.30</td>
  </tr>
  <tr>
    <td>disability</td><td>0.32</td><td>2.68</td><td>2.54</td><td>2.28</td><td>1.88</td><td>2.29</td><td>2.31</td><td>2.74</td>
  </tr>
  <tr>
    <td>nationality</td><td>-0.42</td><td>1.06</td><td>0.96</td><td>0.22</td><td>-0.69</td><td>-0.20</td><td>-0.45</td><td>-0.64</td>
  </tr>
  <tr>
    <td>orientation</td><td>1.84</td><td>1.86</td><td>1.82</td><td>1.77</td><td>1.72</td><td>2.18</td><td>1.59</td><td>1.82</td>
  </tr>
  <tr>
    <td>appearance</td><td>1.19</td><td>0.15</td><td>0.69</td><td>1.47</td><td>1.13</td><td>1.53</td><td>-0.12</td><td>0.27</td>
  </tr>
  <tr>
    <td>age</td><td>0.87</td><td>0.41</td><td>1.23</td><td>1.48</td><td>1.57</td><td>1.64</td><td>0.47</td><td>0.79</td>
  </tr>
  <tr>
    <td rowspan="8">formal stereo/formal anti</td>
    <td>gender</td><td>-0.13</td><td>0.79</td><td>0.44</td><td>1.07</td><td>1.85</td><td>2.29</td><td>0.41</td><td>0.50</td>
  </tr>
  <tr>
    <td>race</td><td>0.22</td><td>1.06</td><td>1.47</td><td>0.91</td><td>2.23</td><td>2.19</td><td>1.05</td><td>1.03</td>
  </tr>
  <tr>
    <td>religion</td><td>0.57</td><td>0.81</td><td>2.47</td><td>0.71</td><td>1.25</td><td>1.67</td><td>0.67</td><td>0.57</td>
  </tr>
  <tr>
    <td>status</td><td>0.29</td><td>1.35</td><td>1.29</td><td>1.23</td><td>1.29</td><td>1.58</td><td>0.89</td><td>1.28</td>
  </tr>
  <tr>
    <td>disability</td><td>1.94</td><td>3.76</td><td>4.30</td><td>2.22</td><td>2.49</td><td>2.06</td><td>3.81</td><td>3.85</td>
  </tr>
  <tr>
    <td>nationality</td><td>0.04</td><td>1.28</td><td>1.84</td><td>1.59</td><td>1.06</td><td>1.47</td><td>0.78</td><td>0.79</td>
  </tr>
  <tr>
    <td>orientation</td><td>2.07</td><td>1.78</td><td>1.35</td><td>1.10</td><td>0.24</td><td>0.36</td><td>1.82</td><td>1.78</td>
  </tr>
  <tr>
    <td>appearance</td><td>0.35</td><td>1.06</td><td>1.84</td><td>0.67</td><td>0.83</td><td>0.99</td><td>0.77</td><td>1.01</td>
  </tr>
  <tr>
    <td>age</td><td>0.40</td><td>1.29</td><td>1.60</td><td>0.99</td><td>1.20</td><td>1.55</td><td>1.28</td><td>1.62</td>
  </tr>
</table>

Table 7: T-value of student’s two-tail test(2).
<table>
  <tr>
    <th>Style</th>
    <th>Bias</th>
    <th>GPT-2</th>
    <th>L-2-7B</th>
    <th>L-2-13B</th>
    <th>OPT-2.7B</th>
    <th>OPT-6.7B</th>
    <th>OPT-13B</th>
    <th>L-3.4-8B</th>
    <th>M-7B</th>
  </tr>
  <tr>
    <td rowspan="10">Original</td>
    <td>gender</td>
    <td>56.87</td><td>59.39</td><td>59.54</td><td>60.09</td><td>62.21</td><td>61.30</td><td>58.68</td><td>55.02</td>
  </tr>
  <tr><td>race</td><td>50.69</td><td>62.08</td><td>63.37</td><td>67.64</td><td>68.69</td><td>64.94</td><td>64.77</td><td>64.77</td></tr>
  <tr><td>religion</td><td>62.86</td><td>75.24</td><td>82.86</td><td>77.14</td><td>76.19</td><td>71.22</td><td>65.74</td><td>69.83</td></tr>
  <tr><td>status</td><td>63.95</td><td>63.95</td><td>70.35</td><td>61.05</td><td>66.28</td><td>64.68</td><td>62.42</td><td>59.65</td></tr>
  <tr><td>disability</td><td>56.67</td><td>83.33</td><td>85.0</td><td>71.67</td><td>75.0</td><td>65.63</td><td>59.48</td><td>62.53</td></tr>
  <tr><td>nationality</td><td>45.91</td><td>60.38</td><td>62.89</td><td>57.86</td><td>63.52</td><td>65.60</td><td>64.4</td><td>64.14</td></tr>
  <tr><td>orientation</td><td>76.19</td><td>73.81</td><td>78.57</td><td>67.86</td><td>67.86</td><td>74.20</td><td>63.43</td><td>73.56</td></tr>
  <tr><td>appearance</td><td>57.14</td><td>68.25</td><td>71.43</td><td>74.6</td><td>71.43</td><td>69.86</td><td>63.67</td><td>66.98</td></tr>
  <tr><td>age</td><td>51.72</td><td>57.56</td><td>70.11</td><td>62.07</td><td>65.52</td><td>77.23</td><td>66.66</td><td>70.11</td></tr>
  <tr><td>AVG</td><td>56.69</td><td>65.55</td><td>67.41</td><td>57.53</td><td>60.55</td><td>68.52</td><td>63.29</td><td>65.24</td></tr>
  <tr>
    <td rowspan="10">Informal</td>
    <td>gender</td>
    <td>49.86</td><td>51.18</td><td>50.15</td><td>54.43</td><td>55.88</td><td>54.47</td><td>51.03</td><td>52.58</td>
  </tr>
  <tr><td>race</td><td>50.89</td><td>54.38</td><td>50.11</td><td>55.56</td><td>55.08</td><td>60.58</td><td>58.47</td><td>63.83</td></tr>
  <tr><td>religion</td><td>51.72</td><td>69.83</td><td>65.63</td><td>65.87</td><td>70.46</td><td>63.10</td><td>56.45</td><td>65.03</td></tr>
  <tr><td>status</td><td>58.32</td><td>59.33</td><td>59.14</td><td>56.61</td><td>61.77</td><td>79.21</td><td>50.78</td><td>53.49</td></tr>
  <tr><td>disability</td><td>61.83</td><td>68.68</td><td>74.06</td><td>60.72</td><td>64.36</td><td>62.20</td><td>56.42</td><td>59.35</td></tr>
  <tr><td>nationality</td><td>44.98</td><td>52.55</td><td>52.96</td><td>53.95</td><td>56.04</td><td>47.30</td><td>55.39</td><td>61.01</td></tr>
  <tr><td>orientation</td><td>57.91</td><td>58.11</td><td>63.89</td><td>61.30</td><td>79.41</td><td>62.64</td><td>56.03</td><td>60.03</td></tr>
  <tr><td>appearance</td><td>53.61</td><td>53.56</td><td>51.73</td><td>51.73</td><td>57.85</td><td>57.85</td><td>57.58</td><td>60.94</td></tr>
  <tr><td>age</td><td>47.89</td><td>56.55</td><td>60.39</td><td>55.36</td><td>65.33</td><td>60.90</td><td>62.42</td><td>68.28</td></tr>
  <tr><td>AVG</td><td>52.42</td><td>58.51</td><td>58.91</td><td>57.46</td><td>59.93</td><td>62.45</td><td>56.15</td><td>61.17</td></tr>
  <tr>
    <td rowspan="10">Formal</td>
    <td>gender</td>
    <td>52.23</td><td>53.89</td><td>60.21</td><td>57.87</td><td>59.67</td><td>53.71</td><td>51.71</td><td>55.0</td>
  </tr>
  <tr><td>race</td><td>46.77</td><td>52.11</td><td>54.39</td><td>55.07</td><td>57.90</td><td>60.88</td><td>57.84</td><td>65.92</td></tr>
  <tr><td>religion</td><td>56.56</td><td>64.35</td><td>65.56</td><td>57.78</td><td>56.12</td><td>64.28</td><td>72.35</td><td>63.6</td></tr>
  <tr><td>status</td><td>59.92</td><td>57.02</td><td>52.98</td><td>57.62</td><td>60.45</td><td>79.40</td><td>52.83</td><td>55.22</td></tr>
  <tr><td>disability</td><td>59.72</td><td>73.33</td><td>79.69</td><td>74.35</td><td>72.90</td><td>60.66</td><td>64.24</td><td>59.29</td></tr>
  <tr><td>nationality</td><td>48.82</td><td>61.16</td><td>65.46</td><td>62.36</td><td>48.63</td><td>61.54</td><td>61.56</td><td>61.56</td></tr>
  <tr><td>orientation</td><td>70.96</td><td>67.76</td><td>75.42</td><td>62.40</td><td>57.88</td><td>81.31</td><td>61.39</td><td>67.96</td></tr>
  <tr><td>appearance</td><td>59.45</td><td>62.59</td><td>59.18</td><td>46.99</td><td>49.06</td><td>59.45</td><td>53.2</td><td>64.72</td></tr>
  <tr><td>age</td><td>58.64</td><td>67.80</td><td>70.28</td><td>64.57</td><td>60.94</td><td>57.99</td><td>74.06</td><td>68.82</td></tr>
  <tr><td>AVG</td><td>56.97</td><td>62.29</td><td>64.26</td><td>59.12</td><td>59.76</td><td>62.37</td><td>59.84</td><td>62.45</td></tr>
  <tr>
    <td rowspan="10">Prefix 1</td>
    <td>gender</td>
    <td>52.67</td><td>61.83</td><td>58.4</td><td>62.21</td><td>62.98</td><td>59.87</td><td>60.52</td><td>60.11</td>
  </tr>
  <tr><td>race</td><td>60.08</td><td>60.27</td><td>63.95</td><td>64.53</td><td>65.89</td><td>64.12</td><td>59.8</td><td>69.74</td></tr>
  <tr><td>religion</td><td>70.48</td><td>75.24</td><td>78.1</td><td>80.95</td><td>78.1</td><td>68.8</td><td>67.38</td><td>67.38</td></tr>
  <tr><td>status</td><td>63.95</td><td>77.19</td><td>64.53</td><td>69.77</td><td>64.02</td><td>62.66</td><td>55.67</td><td>55.67</td></tr>
  <tr><td>disability</td><td>66.67</td><td>78.33</td><td>80.0</td><td>70.0</td><td>73.33</td><td>66.48</td><td>61.24</td><td>59.46</td></tr>
  <tr><td>nationality</td><td>41.51</td><td>64.78</td><td>61.01</td><td>60.38</td><td>62.26</td><td>65.02</td><td>68.41</td><td>60.48</td></tr>
  <tr><td>orientation</td><td>78.57</td><td>71.43</td><td>76.19</td><td>76.19</td><td>79.76</td><td>74.36</td><td>59.4</td><td>70.35</td></tr>
  <tr><td>appearance</td><td>61.9</td><td>73.02</td><td>68.25</td><td>76.19</td><td>74.6</td><td>69.55</td><td>65.69</td><td>65.06</td></tr>
  <tr><td>age</td><td>52.87</td><td>67.82</td><td>67.82</td><td>63.22</td><td>64.37</td><td>78.18</td><td>67.04</td><td>73.82</td></tr>
  <tr><td>AVG</td><td>59.22</td><td>64.79</td><td>66.31</td><td>66.11</td><td>67.31</td><td>67.82</td><td>63.79</td><td>64.67</td></tr>
  <tr>
    <td rowspan="10">Prefix 2</td>
    <td>gender</td>
    <td>53.82</td><td>58.78</td><td>56.49</td><td>63.36</td><td>62.98</td><td>59.21</td><td>58.52</td><td>62.66</td>
  </tr>
  <tr><td>race</td><td>59.41</td><td>60.49</td><td>62.6</td><td>64.53</td><td>66.89</td><td>62.82</td><td>68.27</td><td>68.27</td></tr>
  <tr><td>religion</td><td>70.48</td><td>73.33</td><td>79.05</td><td>78.1</td><td>78.1</td><td>70.9</td><td>65.86</td><td>68.91</td></tr>
  <tr><td>status</td><td>64.53</td><td>66.28</td><td>69.19</td><td>64.53</td><td>69.77</td><td>63.51</td><td>64.18</td><td>57.77</td></tr>
  <tr><td>disability</td><td>63.33</td><td>80.0</td><td>80.0</td><td>75.0</td><td>73.33</td><td>62.73</td><td>60.18</td><td>63.76</td></tr>
  <tr><td>nationality</td><td>44.65</td><td>61.64</td><td>60.38</td><td>61.01</td><td>62.26</td><td>67.34</td><td>63.21</td><td>56.36</td></tr>
  <tr><td>orientation</td><td>77.38</td><td>69.05</td><td>70.24</td><td>78.57</td><td>82.14</td><td>73.11</td><td>59.61</td><td>73.95</td></tr>
  <tr><td>appearance</td><td>66.67</td><td>74.6</td><td>71.43</td><td>73.02</td><td>76.19</td><td>69.85</td><td>65.72</td><td>65.05</td></tr>
  <tr><td>age</td><td>50.57</td><td>71.26</td><td>70.11</td><td>64.37</td><td>64.37</td><td>79.0</td><td>62.46</td><td>74.44</td></tr>
  <tr><td>AVG</td><td>59.79</td><td>64.86</td><td>65.65</td><td>66.04</td><td>67.91</td><td>68.06</td><td>63.45</td><td>65.68</td></tr>
  <tr>
    <td rowspan="10">Prefix 3</td>
    <td>gender</td>
    <td>53.44</td><td>59.16</td><td>56.49</td><td>62.98</td><td>61.07</td><td>60.48</td><td>56.38</td><td>61.25</td>
  </tr>
  <tr><td>race</td><td>59.3</td><td>60.85</td><td>64.15</td><td>64.53</td><td>66.28</td><td>65.36</td><td>58.74</td><td>69.88</td></tr>
  <tr><td>religion</td><td>67.62</td><td>74.29</td><td>79.05</td><td>76.19</td><td>80.0</td><td>71.26</td><td>65.61</td><td>68.66</td></tr>
  <tr><td>status</td><td>62.79</td><td>64.53</td><td>68.6</td><td>63.95</td><td>66.28</td><td>62.24</td><td>66.35</td><td>56.23</td></tr>
  <tr><td>disability</td><td>61.67</td><td>83.33</td><td>81.67</td><td>70.0</td><td>73.33</td><td>63.9</td><td>61.63</td><td>61.1</td></tr>
  <tr><td>nationality</td><td>47.17</td><td>59.75</td><td>59.12</td><td>59.12</td><td>64.15</td><td>66.16</td><td>67.47</td><td>57.16</td></tr>
  <tr><td>orientation</td><td>78.57</td><td>72.41</td><td>72.41</td><td>78.57</td><td>79.76</td><td>73.21</td><td>67.94</td><td>68.85</td></tr>
  <tr><td>appearance</td><td>65.25</td><td>68.25</td><td>73.02</td><td>76.19</td><td>76.19</td><td>68.66</td><td>65.58</td><td>68.26</td></tr>
  <tr><td>age</td><td>51.72</td><td>72.41</td><td>67.82</td><td>62.07</td><td>65.52</td><td>76.86</td><td>63.22</td><td>74.11</td></tr>
  <tr><td>AVG</td><td>59.08</td><td>64.32</td><td>65.65</td><td>65.78</td><td>67.51</td><td>67.57</td><td>63.43</td><td>65.06</td></tr>
  <tr>
    <td rowspan="10">Punctuation</td>
    <td>gender</td>
    <td>54.96</td><td>61.07</td><td>59.92</td><td>60.69</td><td>61.45</td><td>61.16</td><td>60.52</td><td>59.61</td>
  </tr>
  <tr><td>race</td><td>58.91</td><td>62.79</td><td>64.34</td><td>68.02</td><td>69.77</td><td>66.81</td><td>59.8</td><td>69.86</td></tr>
  <tr><td>religion</td><td>60.95</td><td>71.43</td><td>76.19</td><td>77.14</td><td>71.43</td><td>68.45</td><td>67.34</td><td>70.83</td></tr>
  <tr><td>status</td><td>60.47</td><td>63.95</td><td>66.28</td><td>58.72</td><td>65.7</td><td>66.72</td><td>64.66</td><td>55.57</td></tr>
  <tr><td>disability</td><td>56.67</td><td>80.0</td><td>85.0</td><td>71.67</td><td>75.7</td><td>64.99</td><td>61.24</td><td>60.04</td></tr>
  <tr><td>nationality</td><td>47.17</td><td>61.64</td><td>64.15</td><td>55.97</td><td>61.64</td><td>65.83</td><td>60.19</td><td>60.19</td></tr>
  <tr><td>orientation</td><td>76.19</td><td>73.81</td><td>78.57</td><td>67.86</td><td>69.05</td><td>73.86</td><td>59.4</td><td>72.86</td></tr>
  <tr><td>appearance</td><td>57.14</td><td>71.43</td><td>73.02</td><td>71.43</td><td>73.02</td><td>70.2</td><td>65.69</td><td>65.14</td></tr>
  <tr><td>age</td><td>50.57</td><td>71.26</td><td>71.26</td><td>62.07</td><td>65.52</td><td>76.39</td><td>67.04</td><td>70.31</td></tr>
  <tr><td>AVG</td><td>57.36</td><td>65.25</td><td>67.04</td><td>64.99</td><td>67.18</td><td>68.22</td><td>63.79</td><td>64.93</td></tr>
</table>

Table 8: CrowS-Pairs evaluation results (1).
<table>
  <tr>
    <th rowspan="2">Style</th>
    <th rowspan="2">Bias</th>
    <th colspan="6">GP1-2</th>
    <th colspan="6">L-2-13B</th>
    <th colspan="6">OPT-2.7B</th>
    <th colspan="6">OPT-5.7B</th>
    <th colspan="6">OPT-13B</th>
    <th colspan="6">L-3-1.8B</th>
    <th colspan="6">M-7B</th>
  </tr>
  <tr>
    <th>gender</th><th>race</th><th>religion</th><th>status</th><th>disability</th><th>nationality</th><th>orientation</th><th>appearance</th><th>age</th>
    <th>gender</th><th>race</th><th>religion</th><th>status</th><th>disability</th><th>nationality</th><th>orientation</th><th>appearance</th><th>age</th>
    <th>gender</th><th>race</th><th>religion</th><th>status</th><th>disability</th><th>nationality</th><th>orientation</th><th>appearance</th><th>age</th>
    <th>gender</th><th>race</th><th>religion</th><th>status</th><th>disability</th><th>nationality</th><th>orientation</th><th>appearance</th><th>age</th>
    <th>gender</th><th>race</th><th>religion</th><th>status</th><th>disability</th><th>nationality</th><th>orientation</th><th>appearance</th><th>age</th>
    <th>gender</th><th>race</th><th>religion</th><th>status</th><th>disability</th><th>nationality</th><th>orientation</th><th>appearance</th><th>age</th>
  </tr>
  <tr>
    <td rowspan="9">Evident</td>
    <td>gender</td><td>52.82</td><td>59.72</td><td>58.41</td><td>64.43</td><td>61.91</td><td>59.25</td><td>50.29</td><td>65.19</td>
  </tr>
  <tr><td>race</td><td>58.27</td><td>61.51</td><td>62.34</td><td>65.91</td><td>64.16</td><td>66.16</td><td>63.57</td><td>67.18</td></tr>
  <tr><td>religion</td><td>61.65</td><td>71.99</td><td>78.24</td><td>68.67</td><td>73.89</td><td>72.27</td><td>65.28</td><td>68.77</td></tr>
  <tr><td>status</td><td>63.16</td><td>63.57</td><td>68.18</td><td>63.79</td><td>64.44</td><td>64.6</td><td>65.2</td><td>57.27</td></tr>
  <tr><td>disability</td><td>43.19</td><td>81.12</td><td>78.81</td><td>63.26</td><td>65.61</td><td>64.01</td><td>60.51</td><td>64.03</td></tr>
  <tr><td>nationality</td><td>58.87</td><td>60.48</td><td>60.81</td><td>63.69</td><td>62.17</td><td>67.52</td><td>63.08</td><td>65.21</td></tr>
  <tr><td>orientation</td><td>78.22</td><td>68.71</td><td>71.01</td><td>71.00</td><td>78.93</td><td>72.67</td><td>59.15</td><td>74.52</td></tr>
  <tr><td>appearance</td><td>69.91</td><td>64.05</td><td>70.93</td><td>70.16</td><td>73.36</td><td>68.64</td><td>65.41</td><td>56.1</td></tr>
  <tr><td>age</td><td>50.8</td><td>61.64</td><td>67.35</td><td>62.28</td><td>61.6</td><td>69.03</td><td>62.41</td><td>51.8</td></tr>
  <tr>
    <td rowspan="9">QA</td>
    <td>gender</td><td>54.48</td><td>56.51</td><td>57.33</td><td>64.84</td><td>62.44</td><td>58.73</td><td>59.26</td><td>61.33</td>
  </tr>
  <tr><td>race</td><td>60.33</td><td>61.63</td><td>63.29</td><td>62.74</td><td>62.82</td><td>67.29</td><td>64.00</td><td>69.07</td></tr>
  <tr><td>religion</td><td>69.64</td><td>72.57</td><td>76.3</td><td>65.01</td><td>76.42</td><td>69.67</td><td>63.33</td><td>70.22</td></tr>
  <tr><td>status</td><td>63.15</td><td>66.43</td><td>67.75</td><td>76.04</td><td>69.96</td><td>61.71</td><td>65.6</td><td>56.1</td></tr>
  <tr><td>disability</td><td>60.39</td><td>80.5</td><td>58.32</td><td>76.78</td><td>71.11</td><td>63.53</td><td>61.59</td><td>62.4</td></tr>
  <tr><td>nationality</td><td>42.83</td><td>59.77</td><td>78.36</td><td>59.29</td><td>59.85</td><td>66.58</td><td>62.28</td><td>73.46</td></tr>
  <tr><td>orientation</td><td>67.69</td><td>74.37</td><td>69.28</td><td>76.97</td><td>77.36</td><td>74.59</td><td>58.57</td><td>62.61</td></tr>
  <tr><td>appearance</td><td>52.56</td><td>73.18</td><td>68.84</td><td>62.34</td><td>64.11</td><td>78.64</td><td>61.96</td><td>55.89</td></tr>
  <tr><td>AVG</td><td>60.77</td><td>68.04</td><td>67.83</td><td>68.26</td><td>69.29</td><td>67.55</td><td>62.31</td><td>64.97</td></tr>
</table>

Table 9: CrowS-Pairs evaluation results (2).

![Bar charts showing t-values for different models in OPT family](page_101_613_1247_312.png)

Figure 3: T-values from the Student’s two-tailed test of models in OPT family. “original stereo/informal anti” denotes the t-value of the original stereotypical sentence and the informal style anti-stereotypical sentence, and so on.

![Bar charts showing t-values for different models in LLaMA family](page_101_1012_1247_312.png)

Figure 4: T-values from the Student’s two-tailed test of models in LLaMA family. “original stereo/informal anti” denotes the t-value of the original stereotypical sentence and the informal style anti-stereotypical sentence, and so on.
<table>
  <tr>
    <th>Prompt</th>
    <th>Style</th>
    <th>T1-p</th>
    <th>T1-a</th>
    <th>T1-diff</th>
    <th>T2-p</th>
    <th>T2-a</th>
    <th>T2-diff</th>
  </tr>
  <tr>
    <td rowspan="5">(i) "What does p stand for"</td>
    <td>Original</td>
    <td>12.12%</td>
    <td>9.85%</td>
    <td>2.27%</td>
    <td>8.33%</td>
    <td>4.55%</td>
    <td>3.78%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>35.10%</td>
    <td>28.54%</td>
    <td>6.56%</td>
    <td>13.38%</td>
    <td>12.37%</td>
    <td>1.01%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>51.12%</td>
    <td>34.87%</td>
    <td>16.25%</td>
    <td>34.51%</td>
    <td>27.79%</td>
    <td>6.72%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>45.45%</td>
    <td>39.90%</td>
    <td>5.55%</td>
    <td>23.23%</td>
    <td>21.21%</td>
    <td>2.02%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>41.16%</td>
    <td>36.11%</td>
    <td>5.05%</td>
    <td>25.25%</td>
    <td>22.73%</td>
    <td>2.52%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>25.25%</td>
    <td>21.97%</td>
    <td>3.28%</td>
    <td>14.65%</td>
    <td>14.39%</td>
    <td>0.26%</td>
  </tr>
  <tr>
    <td rowspan="5">(ii) "Who or what is/are"</td>
    <td>Original</td>
    <td>41.41%</td>
    <td>31.31%</td>
    <td>10.10%</td>
    <td>25.76%</td>
    <td>20.45%</td>
    <td>5.31%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>44.70%</td>
    <td>35.35%</td>
    <td>9.35%</td>
    <td>23.23%</td>
    <td>18.94%</td>
    <td>4.29%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>49.44%</td>
    <td>37.23%</td>
    <td>12.21%</td>
    <td>31.93%</td>
    <td>27.64%</td>
    <td>4.29%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>53.28%</td>
    <td>39.39%</td>
    <td>13.89%</td>
    <td>33.33%</td>
    <td>26.52%</td>
    <td>6.81%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>47.98%</td>
    <td>37.12%</td>
    <td>10.86%</td>
    <td>35.61%</td>
    <td>27.02%</td>
    <td>8.59%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>52.02%</td>
    <td>39.65%</td>
    <td>12.37%</td>
    <td>36.36%</td>
    <td>29.55%</td>
    <td>6.81%</td>
  </tr>
  <tr>
    <td rowspan="5">(iii) "By p they mean"</td>
    <td>Original</td>
    <td>35.86%</td>
    <td>28.54%</td>
    <td>7.32%</td>
    <td>35.86%</td>
    <td>26.77%</td>
    <td>9.09%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>42.68%</td>
    <td>29.89%</td>
    <td>12.79%</td>
    <td>49.24%</td>
    <td>40.40%</td>
    <td>8.84%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>49.72%</td>
    <td>45.81%</td>
    <td>3.91%</td>
    <td>32.51%</td>
    <td>34.64%</td>
    <td>2.13%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>48.99%</td>
    <td>33.59%</td>
    <td>15.40%</td>
    <td>33.28%</td>
    <td>48.23%</td>
    <td>5.05%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>47.22%</td>
    <td>34.47%</td>
    <td>12.37%</td>
    <td>38.16%</td>
    <td>46.46%</td>
    <td>3.99%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>47.47%</td>
    <td>34.34%</td>
    <td>13.13%</td>
    <td>48.74%</td>
    <td>44.19%</td>
    <td>4.55%</td>
  </tr>
  <tr>
    <td rowspan="5">(iv) "Refers to"</td>
    <td>Original</td>
    <td>41.16%</td>
    <td>41.92%</td>
    <td>0.76%</td>
    <td>46.21%</td>
    <td>45.20%</td>
    <td>1.01%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>47.98%</td>
    <td>50.25%</td>
    <td>2.27%</td>
    <td>29.80%</td>
    <td>31.06%</td>
    <td>1.26%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>38.72%</td>
    <td>43.85%</td>
    <td>5.13%</td>
    <td>44.54%</td>
    <td>49.44%</td>
    <td>4.90%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>47.22%</td>
    <td>34.47%</td>
    <td>12.37%</td>
    <td>38.16%</td>
    <td>46.46%</td>
    <td>3.99%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>48.74%</td>
    <td>47.47%</td>
    <td>1.27%</td>
    <td>32.83%</td>
    <td>35.10%</td>
    <td>2.27%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>45.96%</td>
    <td>46.46%</td>
    <td>0.50%</td>
    <td>39.90%</td>
    <td>40.40%</td>
    <td>0.50%</td>
  </tr>
  <tr>
    <td rowspan="5">(v) "Represent"</td>
    <td>Original</td>
    <td>28.79%</td>
    <td>31.57%</td>
    <td>2.78%</td>
    <td>38.89%</td>
    <td>40.40%</td>
    <td>1.51%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>37.71%</td>
    <td>31.71%</td>
    <td>5.17%</td>
    <td>30.13%</td>
    <td>30.78%</td>
    <td>0.25%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>39.73%</td>
    <td>37.16%</td>
    <td>2.57%</td>
    <td>24.22%</td>
    <td>21.37%</td>
    <td>2.85%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>43.94%</td>
    <td>46.97%</td>
    <td>3.03%</td>
    <td>46.97%</td>
    <td>46.97%</td>
    <td>0%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>44.19%</td>
    <td>48.99%</td>
    <td>4.80%</td>
    <td>47.98%</td>
    <td>47.73%</td>
    <td>0.25%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>40.66%</td>
    <td>41.67%</td>
    <td>1.01%</td>
    <td>47.73%</td>
    <td>46.46%</td>
    <td>1.27%</td>
  </tr>
  <tr>
    <td rowspan="5">(vi) "The pronoun refers to"</td>
    <td>Original</td>
    <td>2.27%</td>
    <td>1.52%</td>
    <td>0.75%</td>
    <td>3.28%</td>
    <td>2.53%</td>
    <td>0.75%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>1.77%</td>
    <td>1.01%</td>
    <td>0.76%</td>
    <td>2.27%</td>
    <td>2.53%</td>
    <td>0.26%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>2.88%</td>
    <td>2.62%</td>
    <td>0.24%</td>
    <td>4.79%</td>
    <td>3.71%</td>
    <td>1.08%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>9.60%</td>
    <td>4.55%</td>
    <td>5.05%</td>
    <td>8.59%</td>
    <td>5.30%</td>
    <td>3.29%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>2.53%</td>
    <td>2.53%</td>
    <td>0.00%</td>
    <td>4.80%</td>
    <td>3.54%</td>
    <td>1.26%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>4.29%</td>
    <td>2.78%</td>
    <td>1.51%</td>
    <td>5.56%</td>
    <td>4.80%</td>
    <td>0.76%</td>
  </tr>
</table>

Table 10: Detailed WinoBias results of GPT-2 across different styles.
<table>
  <tr>
    <th>Prompt</th>
    <th>Style</th>
    <th>T1-p</th>
    <th>T1-a</th>
    <th>T1-diff</th>
    <th>T2-p</th>
    <th>T2-a</th>
    <th>T2-diff</th>
  </tr>
  <tr>
    <td rowspan="6">(i) "What does p stand for"</td>
    <td>Original</td>
    <td>0.25%</td>
    <td>0.00%</td>
    <td>0.25%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>2.27%</td>
    <td>2.27%</td>
    <td>0.00%</td>
    <td>0.50%</td>
    <td>0.76%</td>
    <td>0.26%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>3.38%</td>
    <td>2.09%</td>
    <td>1.29%</td>
    <td>0.80%</td>
    <td>1.74%</td>
    <td>0.94%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>2.02%</td>
    <td>1.26%</td>
    <td>0.76%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td rowspan="6">(ii) "Who or what is/are"</td>
    <td>Original</td>
    <td>6.06%</td>
    <td>3.03%</td>
    <td>3.03%</td>
    <td>1.77%</td>
    <td>0.76%</td>
    <td>1.01%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>3.28%</td>
    <td>2.25%</td>
    <td>1.03%</td>
    <td>0.50%</td>
    <td>0.00%</td>
    <td>0.50%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>2.49%</td>
    <td>2.56%</td>
    <td>0.07%</td>
    <td>4.85%</td>
    <td>3.64%</td>
    <td>1.21%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>25.26%</td>
    <td>37.88%</td>
    <td>12.62%</td>
    <td>2.02%</td>
    <td>0.00%</td>
    <td>2.02%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>2.78%</td>
    <td>35.35%</td>
    <td>32.57%</td>
    <td>1.26%</td>
    <td>0.50%</td>
    <td>0.76%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>2.02%</td>
    <td>1.01%</td>
    <td>1.01%</td>
    <td>0.23%</td>
    <td>0.25%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td rowspan="6">(iii) "By p they mean"</td>
    <td>Original</td>
    <td>75.76%</td>
    <td>41.92%</td>
    <td>33.84%</td>
    <td>86.11%</td>
    <td>64.65%</td>
    <td>21.46%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>68.94%</td>
    <td>37.88%</td>
    <td>31.06%</td>
    <td>79.30%</td>
    <td>55.81%</td>
    <td>23.49%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>74.06%</td>
    <td>53.28%</td>
    <td>20.78%</td>
    <td>90.97%</td>
    <td>87.74%</td>
    <td>3.23%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>79.04%</td>
    <td>40.40%</td>
    <td>38.64%</td>
    <td>88.13%</td>
    <td>77.02%</td>
    <td>11.11%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>80.81%</td>
    <td>45.46%</td>
    <td>33.55%</td>
    <td>12.68%</td>
    <td>8.81%</td>
    <td>10.86%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>78.54%</td>
    <td>44.70%</td>
    <td>33.84%</td>
    <td>89.65%</td>
    <td>73.48%</td>
    <td>16.17%</td>
  </tr>
  <tr>
    <td rowspan="6">(iv) "Refers to"</td>
    <td>Original</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td rowspan="6">(v) "Represent"</td>
    <td>Original</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td rowspan="6">(vi) "The pronoun refers to..."</td>
    <td>Original</td>
    <td>19.44%</td>
    <td>6.82%</td>
    <td>12.62%</td>
    <td>11.11%</td>
    <td>6.06%</td>
    <td>5.05%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>4.04%</td>
    <td>1.26%</td>
    <td>2.78%</td>
    <td>2.52%</td>
    <td>0.51%</td>
    <td>2.01%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>9.61%</td>
    <td>4.54%</td>
    <td>5.07%</td>
    <td>8.39%</td>
    <td>5.27%</td>
    <td>3.12%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>61.36%</td>
    <td>25.76%</td>
    <td>35.60%</td>
    <td>33.84%</td>
    <td>25.76%</td>
    <td>8.08%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>13.38%</td>
    <td>6.82%</td>
    <td>6.56%</td>
    <td>5.30%</td>
    <td>5.05%</td>
    <td>0.25%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>27.27%</td>
    <td>11.62%</td>
    <td>15.65%</td>
    <td>7.32%</td>
    <td>3.54%</td>
    <td>3.78%</td>
  </tr>
</table>

Table 11: Detailed WinoBias results of **LLaMA-2-7B** across different styles.
<table>
  <tr>
    <th>Prompt</th>
    <th>Style</th>
    <th>T1-p</th>
    <th>T1-a</th>
    <th>T1-diff</th>
    <th>T2-p</th>
    <th>T2-a</th>
    <th>T2-diff</th>
  </tr>
  <tr>
    <td rowspan="6">(i) "What does p stand for"</td>
    <td>Original</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>5.69%</td>
    <td>2.92%</td>
    <td>2.77%</td>
    <td>6.18%</td>
    <td>2.73%</td>
    <td>3.45%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>1.01%</td>
    <td>1.26%</td>
    <td>0.25%</td>
    <td>0.00%</td>
    <td>0.25%</td>
    <td>0.25%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>3.03%</td>
    <td>3.54%</td>
    <td>0.51%</td>
    <td>0.25%</td>
    <td>0.00%</td>
    <td>0.25%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>0.76%</td>
    <td>0.51%</td>
    <td>0.25%</td>
    <td>0.25%</td>
    <td>0.00%</td>
    <td>0.25%</td>
  </tr>
  <tr>
    <td rowspan="6">(ii) "Who or what is/are"</td>
    <td>Original</td>
    <td>23.74%</td>
    <td>6.82%</td>
    <td>16.92%</td>
    <td>24.50%</td>
    <td>13.13%</td>
    <td>11.37%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>18.19%</td>
    <td>3.54%</td>
    <td>14.65%</td>
    <td>8.39%</td>
    <td>4.80%</td>
    <td>3.79%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>12.33%</td>
    <td>5.28%</td>
    <td>7.07%</td>
    <td>17.24%</td>
    <td>8.93%</td>
    <td>8.31%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>12.37%</td>
    <td>4.80%</td>
    <td>7.57%</td>
    <td>10.61%</td>
    <td>3.79%</td>
    <td>6.82%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>8.84%</td>
    <td>2.53%</td>
    <td>6.31%</td>
    <td>6.06%</td>
    <td>2.53%</td>
    <td>3.53%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>35.33%</td>
    <td>15.66%</td>
    <td>19.69%</td>
    <td>35.01%</td>
    <td>22.22%</td>
    <td>13.39%</td>
  </tr>
  <tr>
    <td rowspan="6">(iii) "By p they mean"</td>
    <td>Original</td>
    <td>29.80%</td>
    <td>17.68%</td>
    <td>12.12%</td>
    <td>30.30%</td>
    <td>29.80%</td>
    <td>0.50%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>42.42%</td>
    <td>25.76%</td>
    <td>16.66%</td>
    <td>32.78%</td>
    <td>30.76%</td>
    <td>2.02%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>66.71%</td>
    <td>43.99%</td>
    <td>22.72%</td>
    <td>79.65%</td>
    <td>69.08%</td>
    <td>10.57%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>58.33%</td>
    <td>37.63%</td>
    <td>20.70%</td>
    <td>57.83%</td>
    <td>59.34%</td>
    <td>1.51%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>67.18%</td>
    <td>43.94%</td>
    <td>23.24%</td>
    <td>79.08%</td>
    <td>73.26%</td>
    <td>6.38%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>70.96%</td>
    <td>44.70%</td>
    <td>26.26%</td>
    <td>79.80%</td>
    <td>73.74%</td>
    <td>6.06%</td>
  </tr>
  <tr>
    <td rowspan="6">(iv) "Refers to"</td>
    <td>Original</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td rowspan="6">(v) "Represent"</td>
    <td>Original</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td rowspan="6">(vi) "The pronoun refers to"</td>
    <td>Original</td>
    <td>12.63%</td>
    <td>5.30%</td>
    <td>7.33%</td>
    <td>16.67%</td>
    <td>8.59%</td>
    <td>8.08%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>5.56%</td>
    <td>2.78%</td>
    <td>2.78%</td>
    <td>6.31%</td>
    <td>2.78%</td>
    <td>3.53%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>26.87%</td>
    <td>11.73%</td>
    <td>15.14%</td>
    <td>7.14%</td>
    <td>3.59%</td>
    <td>3.55%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>22.23%</td>
    <td>11.36%</td>
    <td>11.87%</td>
    <td>21.82%</td>
    <td>21.46%</td>
    <td>10.36%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>17.18%</td>
    <td>9.60%</td>
    <td>7.58%</td>
    <td>18.19%</td>
    <td>13.89%</td>
    <td>4.30%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>28.28%</td>
    <td>14.65%</td>
    <td>13.63%</td>
    <td>42.93%</td>
    <td>28.54%</td>
    <td>14.39%</td>
  </tr>
</table>

Table 12: Detailed WinoBias results of **LLaMA-2-13B** across different styles.
<table>
  <tr>
    <th>Prompt</th>
    <th>Style</th>
    <th>T1-p</th>
    <th>T1-a</th>
    <th>T1-diff</th>
    <th>T2-p</th>
    <th>T2-a</th>
    <th>T2-diff</th>
  </tr>
  <tr>
    <td rowspan="6">(i) "What does p stand for"</td>
    <td>Original</td>
    <td>6.82%</td>
    <td>2.78%</td>
    <td>4.04%</td>
    <td>6.57%</td>
    <td>3.03%</td>
    <td>3.54%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>12.63%</td>
    <td>4.04%</td>
    <td>8.59%</td>
    <td>15.40%</td>
    <td>7.83%</td>
    <td>7.57%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>26.87%</td>
    <td>14.09%</td>
    <td>12.78%</td>
    <td>26.09%</td>
    <td>15.25%</td>
    <td>10.84%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>54.04%</td>
    <td>29.04%</td>
    <td>25.00%</td>
    <td>50.00%</td>
    <td>26.26%</td>
    <td>23.74%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>46.47%</td>
    <td>25.00%</td>
    <td>21.47%</td>
    <td>41.41%</td>
    <td>20.71%</td>
    <td>20.70%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>28.03%</td>
    <td>11.87%</td>
    <td>16.16%</td>
    <td>29.29%</td>
    <td>14.39%</td>
    <td>14.90%</td>
  </tr>
  <tr>
    <td rowspan="6">(ii) "Who or what is/are"</td>
    <td>Original</td>
    <td>22.98%</td>
    <td>9.60%</td>
    <td>13.38%</td>
    <td>7.58%</td>
    <td>3.79%</td>
    <td>3.79%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>17.68%</td>
    <td>6.57%</td>
    <td>11.11%</td>
    <td>6.31%</td>
    <td>3.28%</td>
    <td>3.03%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>40.58%</td>
    <td>18.19%</td>
    <td>22.39%</td>
    <td>20.50%</td>
    <td>13.27%</td>
    <td>7.23%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>43.43%</td>
    <td>25.25%</td>
    <td>18.18%</td>
    <td>17.42%</td>
    <td>7.82%</td>
    <td>9.60%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>40.15%</td>
    <td>19.19%</td>
    <td>20.96%</td>
    <td>18.19%</td>
    <td>7.83%</td>
    <td>10.36%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>35.10%</td>
    <td>17.42%</td>
    <td>17.68%</td>
    <td>16.92%</td>
    <td>7.32%</td>
    <td>9.60%</td>
  </tr>
  <tr>
    <td rowspan="6">(iii) "By p they mean"</td>
    <td>Original</td>
    <td>31.06%</td>
    <td>21.46%</td>
    <td>9.60%</td>
    <td>65.15%</td>
    <td>51.01%</td>
    <td>14.14%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>35.61%</td>
    <td>21.72%</td>
    <td>13.89%</td>
    <td>60.61%</td>
    <td>50.00%</td>
    <td>10.61%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>43.34%</td>
    <td>18.28%</td>
    <td>25.06%</td>
    <td>15.50%</td>
    <td>7.64%</td>
    <td>7.86%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>56.31%</td>
    <td>31.57%</td>
    <td>24.74%</td>
    <td>78.28%</td>
    <td>58.33%</td>
    <td>19.95%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>56.62%</td>
    <td>18.69%</td>
    <td>17.93%</td>
    <td>74.66%</td>
    <td>44.44%</td>
    <td>16.42%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>52.07%</td>
    <td>20.96%</td>
    <td>11.11%</td>
    <td>73.23%</td>
    <td>57.07%</td>
    <td>16.16%</td>
  </tr>
  <tr>
    <td rowspan="6">(iv) "Refers to"</td>
    <td>Original</td>
    <td>41.92%</td>
    <td>38.13%</td>
    <td>3.79%</td>
    <td>15.15%</td>
    <td>13.13%</td>
    <td>2.02%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>41.41%</td>
    <td>32.58%</td>
    <td>8.83%</td>
    <td>11.11%</td>
    <td>9.34%</td>
    <td>1.77%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>56.64%</td>
    <td>42.38%</td>
    <td>14.30%</td>
    <td>13.34%</td>
    <td>9.07%</td>
    <td>4.27%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>56.31%</td>
    <td>31.57%</td>
    <td>24.74%</td>
    <td>78.28%</td>
    <td>58.33%</td>
    <td>19.95%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>46.47%</td>
    <td>42.68%</td>
    <td>3.79%</td>
    <td>5.81%</td>
    <td>5.30%</td>
    <td>0.51%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>47.47%</td>
    <td>42.68%</td>
    <td>4.79%</td>
    <td>11.87%</td>
    <td>8.84%</td>
    <td>3.03%</td>
  </tr>
  <tr>
    <td rowspan="6">(v) "Represent"</td>
    <td>Original</td>
    <td>41.16%</td>
    <td>35.86%</td>
    <td>5.30%</td>
    <td>39.00%</td>
    <td>35.55%</td>
    <td>3.55%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>38.57%</td>
    <td>30.78%</td>
    <td>7.79%</td>
    <td>24.35%</td>
    <td>20.97%</td>
    <td>3.38%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>56.57%</td>
    <td>41.00%</td>
    <td>15.57%</td>
    <td>18.32%</td>
    <td>16.59%</td>
    <td>1.73%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>36.87%</td>
    <td>33.84%</td>
    <td>5.03%</td>
    <td>15.40%</td>
    <td>12.37%</td>
    <td>3.03%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>30.56%</td>
    <td>25.51%</td>
    <td>5.05%</td>
    <td>11.62%</td>
    <td>8.08%</td>
    <td>3.54%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>32.32%</td>
    <td>28.54%</td>
    <td>3.78%</td>
    <td>18.69%</td>
    <td>16.67%</td>
    <td>2.02%</td>
  </tr>
  <tr>
    <td rowspan="6">(vi) "The pronoun refers to"</td>
    <td>Original</td>
    <td>71.97%</td>
    <td>29.30%</td>
    <td>42.67%</td>
    <td>81.57%</td>
    <td>55.30%</td>
    <td>26.27%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>72.98%</td>
    <td>27.28%</td>
    <td>45.70%</td>
    <td>83.08%</td>
    <td>56.57%</td>
    <td>26.51%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>84.82%</td>
    <td>26.22%</td>
    <td>58.60%</td>
    <td>89.35%</td>
    <td>61.50%</td>
    <td>27.85%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>77.78%</td>
    <td>27.02%</td>
    <td>50.76%</td>
    <td>85.10%</td>
    <td>56.82%</td>
    <td>28.28%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>74.24%</td>
    <td>27.78%</td>
    <td>46.46%</td>
    <td>85.86%</td>
    <td>57.58%</td>
    <td>28.28%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>77.78%</td>
    <td>29.80%</td>
    <td>47.98%</td>
    <td>83.59%</td>
    <td>55.81%</td>
    <td>27.78%</td>
  </tr>
</table>

Table 13: Detailed WinoBias results of OPT-2.7B across different styles.
<table>
  <tr>
    <th>Prompt</th>
    <th>Style</th>
    <th>T1-p</th>
    <th>T1-a</th>
    <th>T1-diff</th>
    <th>T2-p</th>
    <th>T2-a</th>
    <th>T2-diff</th>
  </tr>
  <tr>
    <td rowspan="5">(i) "What does p stand for"</td>
    <td>Original</td>
    <td>25.00%</td>
    <td>18.94%</td>
    <td>6.06%</td>
    <td>32.57%</td>
    <td>17.42%</td>
    <td>15.15%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>45.45%</td>
    <td>21.97%</td>
    <td>23.48%</td>
    <td>60.35%</td>
    <td>34.09%</td>
    <td>26.26%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>53.97%</td>
    <td>32.70%</td>
    <td>21.27%</td>
    <td>46.98%</td>
    <td>30.11%</td>
    <td>16.87%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>56.31%</td>
    <td>40.15%</td>
    <td>16.16%</td>
    <td>41.16%</td>
    <td>23.99%</td>
    <td>17.17%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>56.06%</td>
    <td>33.84%</td>
    <td>22.22%</td>
    <td>47.98%</td>
    <td>25.76%</td>
    <td>22.22%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>52.27%</td>
    <td>33.58%</td>
    <td>18.69%</td>
    <td>48.48%</td>
    <td>27.78%</td>
    <td>20.70%</td>
  </tr>
  <tr>
    <td rowspan="5">(ii) "Who or what is/are"</td>
    <td>Original</td>
    <td>41.92%</td>
    <td>24.24%</td>
    <td>17.68%</td>
    <td>17.42%</td>
    <td>5.56%</td>
    <td>11.86%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>44.95%</td>
    <td>21.46%</td>
    <td>23.49%</td>
    <td>23.74%</td>
    <td>10.10%</td>
    <td>13.64%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>67.03%</td>
    <td>28.05%</td>
    <td>38.98%</td>
    <td>45.34%</td>
    <td>14.70%</td>
    <td>30.64%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>61.87%</td>
    <td>38.38%</td>
    <td>23.49%</td>
    <td>34.85%</td>
    <td>15.40%</td>
    <td>19.45%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>63.89%</td>
    <td>34.85%</td>
    <td>29.04%</td>
    <td>44.44%</td>
    <td>18.69%</td>
    <td>25.75%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>63.38%</td>
    <td>34.34%</td>
    <td>29.04%</td>
    <td>39.40%</td>
    <td>15.91%</td>
    <td>23.49%</td>
  </tr>
  <tr>
    <td rowspan="5">(iii) "By p they mean..."</td>
    <td>Original</td>
    <td>40.15%</td>
    <td>28.03%</td>
    <td>12.12%</td>
    <td>73.74%</td>
    <td>61.36%</td>
    <td>12.38%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>50.51%</td>
    <td>33.59%</td>
    <td>16.92%</td>
    <td>77.53%</td>
    <td>66.16%</td>
    <td>11.37%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>41.33%</td>
    <td>32.82%</td>
    <td>8.51%</td>
    <td>80.03%</td>
    <td>52.84%</td>
    <td>27.19%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>72.22%</td>
    <td>38.89%</td>
    <td>33.33%</td>
    <td>88.89%</td>
    <td>67.55%</td>
    <td>20.96%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>66.67%</td>
    <td>34.85%</td>
    <td>31.82%</td>
    <td>85.86%</td>
    <td>64.56%</td>
    <td>20.96%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>61.36%</td>
    <td>35.86%</td>
    <td>25.50%</td>
    <td>83.33%</td>
    <td>69.44%</td>
    <td>13.89%</td>
  </tr>
  <tr>
    <td rowspan="5">(iv) "Refers to"</td>
    <td>Original</td>
    <td>14.65%</td>
    <td>11.11%</td>
    <td>3.54%</td>
    <td>6.06%</td>
    <td>4.55%</td>
    <td>1.51%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>3.54%</td>
    <td>1.52%</td>
    <td>2.02%</td>
    <td>1.52%</td>
    <td>1.01%</td>
    <td>0.51%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>23.47%</td>
    <td>20.96%</td>
    <td>2.51%</td>
    <td>8.00%</td>
    <td>3.63%</td>
    <td>4.37%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>11.11%</td>
    <td>7.83%</td>
    <td>3.28%</td>
    <td>1.26%</td>
    <td>2.02%</td>
    <td>0.76%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>11.11%</td>
    <td>7.83%</td>
    <td>3.28%</td>
    <td>1.26%</td>
    <td>2.02%</td>
    <td>0.76%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>25.00%</td>
    <td>20.45%</td>
    <td>4.55%</td>
    <td>7.58%</td>
    <td>3.79%</td>
    <td>3.79%</td>
  </tr>
  <tr>
    <td rowspan="5">(v) "Represent"</td>
    <td>Original</td>
    <td>49.75%</td>
    <td>46.72%</td>
    <td>3.03%</td>
    <td>12.12%</td>
    <td>11.36%</td>
    <td>0.76%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>33.59%</td>
    <td>30.03%</td>
    <td>3.56%</td>
    <td>13.47%</td>
    <td>13.47%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>20.96%</td>
    <td>17.22%</td>
    <td>3.74%</td>
    <td>9.83%</td>
    <td>7.83%</td>
    <td>2.00%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>45.96%</td>
    <td>41.92%</td>
    <td>4.04%</td>
    <td>9.85%</td>
    <td>8.59%</td>
    <td>1.26%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>40.91%</td>
    <td>36.62%</td>
    <td>4.29%</td>
    <td>9.85%</td>
    <td>9.09%</td>
    <td>0.76%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>44.95%</td>
    <td>40.90%</td>
    <td>4.05%</td>
    <td>12.37%</td>
    <td>8.59%</td>
    <td>3.78%</td>
  </tr>
  <tr>
    <td rowspan="5">(vi) "The pronoun refers to"</td>
    <td>Original</td>
    <td>78.03%</td>
    <td>32.83%</td>
    <td>45.20%</td>
    <td>82.07%</td>
    <td>45.71%</td>
    <td>36.36%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>82.07%</td>
    <td>30.56%</td>
    <td>51.51%</td>
    <td>87.88%</td>
    <td>55.31%</td>
    <td>32.07%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>80.83%</td>
    <td>37.68%</td>
    <td>43.15%</td>
    <td>72.22%</td>
    <td>48.09%</td>
    <td>24.13%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>80.56%</td>
    <td>31.57%</td>
    <td>48.99%</td>
    <td>81.31%</td>
    <td>45.45%</td>
    <td>35.86%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>80.05%</td>
    <td>30.05%</td>
    <td>50.00%</td>
    <td>87.63%</td>
    <td>49.50%</td>
    <td>38.13%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>81.06%</td>
    <td>33.08%</td>
    <td>47.98%</td>
    <td>85.86%</td>
    <td>49.50%</td>
    <td>36.36%</td>
  </tr>
</table>

Table 14: Detailed WinoBias results of OPT-6.7B across different styles.
<table>
  <tr>
    <th>Prompt</th>
    <th>Style</th>
    <th>T1-p</th>
    <th>T1-a</th>
    <th>T1-diff</th>
    <th>T2-p</th>
    <th>T2-a</th>
    <th>T2-diff</th>
  </tr>
  <tr>
    <td rowspan="5">(i) "What does p stand for"</td>
    <td>Original</td>
    <td>47.22%</td>
    <td>30.05%</td>
    <td>17.17%</td>
    <td>76.77%</td>
    <td>63.38%</td>
    <td>13.39%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>59.60%</td>
    <td>27.27%</td>
    <td>32.33%</td>
    <td>61.62%</td>
    <td>36.11%</td>
    <td>25.51%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>66.70%</td>
    <td>31.15%</td>
    <td>35.55%</td>
    <td>64.18%</td>
    <td>35.75%</td>
    <td>28.43%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>67.70%</td>
    <td>29.16%</td>
    <td>38.53%</td>
    <td>58.33%</td>
    <td>36.74%</td>
    <td>21.59%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>63.45%</td>
    <td>32.34%</td>
    <td>31.11%</td>
    <td>62.86%</td>
    <td>42.65%</td>
    <td>20.21%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>61.37%</td>
    <td>24.77%</td>
    <td>36.60%</td>
    <td>66.27%</td>
    <td>40.66%</td>
    <td>25.61%</td>
  </tr>
  <tr>
    <td rowspan="5">(ii) "Who or what is/are"</td>
    <td>Original</td>
    <td>15.66%</td>
    <td>10.10%</td>
    <td>5.56%</td>
    <td>10.86%</td>
    <td>8.33%</td>
    <td>2.53%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>38.89%</td>
    <td>15.40%</td>
    <td>23.49%</td>
    <td>21.72%</td>
    <td>10.86%</td>
    <td>10.86%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>43.76%</td>
    <td>17.45%</td>
    <td>26.31%</td>
    <td>25.55%</td>
    <td>10.80%</td>
    <td>14.75%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>39.27%</td>
    <td>16.28%</td>
    <td>23.00%</td>
    <td>21.22%</td>
    <td>10.01%</td>
    <td>11.21%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>40.66%</td>
    <td>16.22%</td>
    <td>24.43%</td>
    <td>25.09%</td>
    <td>10.16%</td>
    <td>14.94%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>42.22%</td>
    <td>15.21%</td>
    <td>27.01%</td>
    <td>21.60%</td>
    <td>12.09%</td>
    <td>9.51%</td>
  </tr>
  <tr>
    <td rowspan="5">(iii) "By p they mean.."</td>
    <td>Original</td>
    <td>26.26%</td>
    <td>19.70%</td>
    <td>6.56%</td>
    <td>70.71%</td>
    <td>60.10%</td>
    <td>10.61%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>38.38%</td>
    <td>22.73%</td>
    <td>15.65%</td>
    <td>58.08%</td>
    <td>43.69%</td>
    <td>14.39%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>38.24%</td>
    <td>23.55%</td>
    <td>14.69%</td>
    <td>61.87%</td>
    <td>43.09%</td>
    <td>18.78%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>40.42%</td>
    <td>22.19%</td>
    <td>18.24%</td>
    <td>59.96%</td>
    <td>39.48%</td>
    <td>20.47%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>38.36%</td>
    <td>21.71%</td>
    <td>14.61%</td>
    <td>64.09%</td>
    <td>39.46%</td>
    <td>24.58%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>42.16%</td>
    <td>21.82%</td>
    <td>20.34%</td>
    <td>68.31%</td>
    <td>50.46%</td>
    <td>17.85%</td>
  </tr>
  <tr>
    <td rowspan="5">(iv) "Refers to"</td>
    <td>Original</td>
    <td>14.65%</td>
    <td>16.41%</td>
    <td>1.76%</td>
    <td>2.27%</td>
    <td>2.25%</td>
    <td>0.02%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>5.56%</td>
    <td>2.78%</td>
    <td>2.78%</td>
    <td>1.26%</td>
    <td>0.25%</td>
    <td>1.01%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>5.39%</td>
    <td>2.34%</td>
    <td>3.42%</td>
    <td>1.14%</td>
    <td>0.25%</td>
    <td>0.89%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>5.90%</td>
    <td>2.86%</td>
    <td>3.04%</td>
    <td>1.37%</td>
    <td>0.32%</td>
    <td>1.12%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>6.19%</td>
    <td>2.54%</td>
    <td>3.65%</td>
    <td>1.41%</td>
    <td>0.25%</td>
    <td>1.19%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>5.04%</td>
    <td>2.57%</td>
    <td>2.48%</td>
    <td>1.18%</td>
    <td>0.25%</td>
    <td>0.93%</td>
  </tr>
  <tr>
    <td rowspan="5">(v) "Represent"</td>
    <td>Original</td>
    <td>40.40%</td>
    <td>40.40%</td>
    <td>0.00%</td>
    <td>7.58%</td>
    <td>7.83%</td>
    <td>0.25%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>31.06%</td>
    <td>29.70%</td>
    <td>1.36%</td>
    <td>13.05%</td>
    <td>13.19%</td>
    <td>2.77%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>31.72%</td>
    <td>27.12%</td>
    <td>4.60%</td>
    <td>7.06%</td>
    <td>14.35%</td>
    <td>7.29%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>31.55%</td>
    <td>25.23%</td>
    <td>6.32%</td>
    <td>6.23%</td>
    <td>15.67%</td>
    <td>0.56%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>29.32%</td>
    <td>24.98%</td>
    <td>4.34%</td>
    <td>15.40%</td>
    <td>13.68%</td>
    <td>1.72%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>29.30%</td>
    <td>25.56%</td>
    <td>3.74%</td>
    <td>15.71%</td>
    <td>14.53%</td>
    <td>1.18%</td>
  </tr>
  <tr>
    <td rowspan="5">(vi) "The pronoun refers to"</td>
    <td>Original</td>
    <td>68.69%</td>
    <td>32.58%</td>
    <td>36.11%</td>
    <td>93.18%</td>
    <td>77.53%</td>
    <td>15.65%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>80.56%</td>
    <td>28.03%</td>
    <td>52.53%</td>
    <td>85.35%</td>
    <td>49.49%</td>
    <td>35.86%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>79.00%</td>
    <td>27.06%</td>
    <td>55.95%</td>
    <td>72.45%</td>
    <td>40.31%</td>
    <td>32.14%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>87.70%</td>
    <td>31.06%</td>
    <td>56.65%</td>
    <td>90.62%</td>
    <td>55.20%</td>
    <td>35.42%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>84.59%</td>
    <td>25.70%</td>
    <td>58.89%</td>
    <td>87.49%</td>
    <td>50.12%</td>
    <td>37.37%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>83.50%</td>
    <td>29.23%</td>
    <td>54.27%</td>
    <td>75.81%</td>
    <td>40.42%</td>
    <td>35.39%</td>
  </tr>
</table>

Table 15: Detailed WinoBias results of OPT-13B across different styles.
<table>
  <tr>
    <th>Prompt</th>
    <th>Style</th>
    <th>T1-p</th>
    <th>T1-a</th>
    <th>T1-diff</th>
    <th>T2-p</th>
    <th>T2-a</th>
    <th>T2-diff</th>
  </tr>
  <tr>
    <td rowspan="6">(i) "What does p stand for"</td>
    <td>Original</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>0.25%</td>
    <td>0.25%</td>
    <td>0.00%</td>
    <td>0.25%</td>
    <td>0.25%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>0.25%</td>
    <td>0.17%</td>
    <td>0.09%</td>
    <td>0.23%</td>
    <td>0.17%</td>
    <td>0.06%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td rowspan="6">(ii) "Who or what is/are"</td>
    <td>Original</td>
    <td>11.87%</td>
    <td>9.34%</td>
    <td>2.53%</td>
    <td>2.53%</td>
    <td>1.01%</td>
    <td>1.52%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>13.64%</td>
    <td>9.85%</td>
    <td>3.79%</td>
    <td>3.03%</td>
    <td>1.26%</td>
    <td>1.77%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>14.80%</td>
    <td>9.88%</td>
    <td>4.91%</td>
    <td>2.77%</td>
    <td>1.25%</td>
    <td>1.52%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>14.62%</td>
    <td>10.09%</td>
    <td>4.53%</td>
    <td>3.28%</td>
    <td>1.32%</td>
    <td>1.96%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>12.87%</td>
    <td>9.66%</td>
    <td>3.21%</td>
    <td>3.05%</td>
    <td>1.20%</td>
    <td>1.84%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>13.60%</td>
    <td>10.62%</td>
    <td>2.98%</td>
    <td>3.14%</td>
    <td>1.26%</td>
    <td>1.88%</td>
  </tr>
  <tr>
    <td rowspan="6">(iii) "By p they mean"</td>
    <td>Original</td>
    <td>50.76%</td>
    <td>29.80%</td>
    <td>20.96%</td>
    <td>65.66%</td>
    <td>48.74%</td>
    <td>16.92%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>58.08%</td>
    <td>29.29%</td>
    <td>28.79%</td>
    <td>70.70%</td>
    <td>54.29%</td>
    <td>16.41%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>59.85%</td>
    <td>30.02%</td>
    <td>29.83%</td>
    <td>75.42%</td>
    <td>50.90%</td>
    <td>24.51%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>59.72%</td>
    <td>31.92%</td>
    <td>27.80%</td>
    <td>68.52%</td>
    <td>51.45%</td>
    <td>17.06%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>58.10%</td>
    <td>29.66%</td>
    <td>24.45%</td>
    <td>71.58%</td>
    <td>56.43%</td>
    <td>14.20%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>61.26%</td>
    <td>30.30%</td>
    <td>30.96%</td>
    <td>68.30%</td>
    <td>53.50%</td>
    <td>14.80%</td>
  </tr>
  <tr>
    <td rowspan="6">(iv) "Refers to"</td>
    <td>Original</td>
    <td>7.83%</td>
    <td>3.79%</td>
    <td>4.04%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td rowspan="6">(v) "Represent"</td>
    <td>Original</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td rowspan="6">(vi) "The pronoun refers to"</td>
    <td>Original</td>
    <td>77.27%</td>
    <td>36.87%</td>
    <td>40.40%</td>
    <td>85.35%</td>
    <td>64.39%</td>
    <td>20.96%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>63.38%</td>
    <td>26.26%</td>
    <td>37.12%</td>
    <td>76.01%</td>
    <td>54.54%</td>
    <td>21.47%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>68.71%</td>
    <td>25.16%</td>
    <td>41.55%</td>
    <td>79.40%</td>
    <td>53.79%</td>
    <td>25.61%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>66.16%</td>
    <td>26.29%</td>
    <td>39.87%</td>
    <td>77.42%</td>
    <td>57.50%</td>
    <td>19.93%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>68.24%</td>
    <td>24.82%</td>
    <td>43.42%</td>
    <td>73.68%</td>
    <td>58.38%</td>
    <td>15.30%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>68.40%</td>
    <td>28.32%</td>
    <td>40.08%</td>
    <td>75.98%</td>
    <td>51.57%</td>
    <td>24.41%</td>
  </tr>
</table>

Table 16: Detailed WinoBias results of Mitral-7B across different styles.
<table>
  <tr>
    <th>Prompt</th>
    <th>Style</th>
    <th>T1-p</th>
    <th>T1-a</th>
    <th>T1-diff</th>
    <th>T2-p</th>
    <th>T2-a</th>
    <th>T2-diff</th>
  </tr>
  <tr>
    <td rowspan="5">(i) "What does p stand for"</td>
    <td>Original</td>
    <td>22.47%</td>
    <td>12.37%</td>
    <td>10.10%</td>
    <td>24.49%</td>
    <td>17.93%</td>
    <td>6.56%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>46.97%</td>
    <td>22.98%</td>
    <td>23.99%</td>
    <td>43.18%</td>
    <td>28.28%</td>
    <td>14.90%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>21.67%</td>
    <td>12.17%</td>
    <td>9.50%</td>
    <td>23.98%</td>
    <td>18.05%</td>
    <td>5.93%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>30.17%</td>
    <td>17.04%</td>
    <td>13.12%</td>
    <td>25.33%</td>
    <td>18.55%</td>
    <td>6.78%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>26.93%</td>
    <td>16.40%</td>
    <td>10.52%</td>
    <td>29.29%</td>
    <td>25.05%</td>
    <td>4.24%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>26.40%</td>
    <td>16.23%</td>
    <td>10.17%</td>
    <td>21.14%</td>
    <td>11.95%</td>
    <td>9.19%</td>
  </tr>
  <tr>
    <td rowspan="5">(ii) "Who or what is/are"</td>
    <td>Original</td>
    <td>86.62%</td>
    <td>37.88%</td>
    <td>48.74%</td>
    <td>94.19%</td>
    <td>85.10%</td>
    <td>9.09%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>86.11%</td>
    <td>37.88%</td>
    <td>48.23%</td>
    <td>95.71%</td>
    <td>82.32%</td>
    <td>13.39%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>80.54%</td>
    <td>38.88%</td>
    <td>41.65%</td>
    <td>92.62%</td>
    <td>77.64%</td>
    <td>14.98%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>88.87%</td>
    <td>36.27%</td>
    <td>52.60%</td>
    <td>92.99%</td>
    <td>82.48%</td>
    <td>10.52%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>88.36%</td>
    <td>39.07%</td>
    <td>49.29%</td>
    <td>93.68%</td>
    <td>81.84%</td>
    <td>11.84%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>83.70%</td>
    <td>36.21%</td>
    <td>47.49%</td>
    <td>97.57%</td>
    <td>85.88%</td>
    <td>11.69%</td>
  </tr>
  <tr>
    <td rowspan="5">(iii) "By p they mean.."</td>
    <td>Original</td>
    <td>83.08%</td>
    <td>46.21%</td>
    <td>36.87%</td>
    <td>89.90%</td>
    <td>74.24%</td>
    <td>15.66%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>84.60%</td>
    <td>45.71%</td>
    <td>38.89%</td>
    <td>92.42%</td>
    <td>77.02%</td>
    <td>15.40%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>87.49%</td>
    <td>48.16%</td>
    <td>39.34%</td>
    <td>85.45%</td>
    <td>73.83%</td>
    <td>11.62%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>89.74%</td>
    <td>48.53%</td>
    <td>41.21%</td>
    <td>92.02%</td>
    <td>77.13%</td>
    <td>14.79%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>86.76%</td>
    <td>49.14%</td>
    <td>37.45%</td>
    <td>94.48%</td>
    <td>77.66%</td>
    <td>16.81%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>81.03%</td>
    <td>48.14%</td>
    <td>32.89%</td>
    <td>89.96%</td>
    <td>73.75%</td>
    <td>16.21%</td>
  </tr>
  <tr>
    <td rowspan="5">(iv) "Refers to"</td>
    <td>Original</td>
    <td>0.50%</td>
    <td>1.52%</td>
    <td>1.02%</td>
    <td>0.50%</td>
    <td>0.25%</td>
    <td>0.25%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>0.50%</td>
    <td>1.53%</td>
    <td>1.03%</td>
    <td>0.49%</td>
    <td>0.26%</td>
    <td>0.23%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>0.50%</td>
    <td>1.53%</td>
    <td>1.03%</td>
    <td>0.49%</td>
    <td>0.26%</td>
    <td>0.23%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>0.50%</td>
    <td>1.46%</td>
    <td>0.97%</td>
    <td>0.52%</td>
    <td>0.25%</td>
    <td>0.27%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>0.49%</td>
    <td>1.53%</td>
    <td>1.05%</td>
    <td>0.49%</td>
    <td>0.24%</td>
    <td>0.25%</td>
  </tr>
  <tr>
    <td rowspan="5">(v) "Represent"</td>
    <td>Original</td>
    <td>0.25%</td>
    <td>0.50%</td>
    <td>0.25%</td>
    <td>0.00%</td>
    <td>0.25%</td>
    <td>0.25%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>0.00%</td>
  </tr>
  <tr>
    <td rowspan="5">(vi) "The pronoun refers to"</td>
    <td>Original</td>
    <td>83.33%</td>
    <td>36.87%</td>
    <td>46.46%</td>
    <td>94.70%</td>
    <td>78.03%</td>
    <td>16.67%</td>
  </tr>
  <tr>
    <td>Evident</td>
    <td>79.04%</td>
    <td>34.09%</td>
    <td>44.95%</td>
    <td>90.91%</td>
    <td>77.02%</td>
    <td>13.89%</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>75.56%</td>
    <td>35.70%</td>
    <td>39.87%</td>
    <td>89.19%</td>
    <td>73.18%</td>
    <td>16.01%</td>
  </tr>
  <tr>
    <td>Prefix1</td>
    <td>78.68%</td>
    <td>32.75%</td>
    <td>43.93%</td>
    <td>88.37%</td>
    <td>79.15%</td>
    <td>9.23%</td>
  </tr>
  <tr>
    <td>Prefix2</td>
    <td>75.74%</td>
    <td>33.47%</td>
    <td>42.27%</td>
    <td>92.98%</td>
    <td>74.25%</td>
    <td>18.73%</td>
  </tr>
  <tr>
    <td>Prefix3</td>
    <td>76.76%</td>
    <td>32.46%</td>
    <td>44.30%</td>
    <td>93.47%</td>
    <td>73.82%</td>
    <td>19.66%</td>
  </tr>
</table>

Table 17: Detailed WinoBias results of **LLaMA-3.1-8B** across different styles.

Reviewer paper summaries: Authors show that bias metrics can be influenced by stylistic changes while the underlying content remains consistent. They also propose a new framework with causal inference to find to what degree style and content influence these metrics with different types of bias.
Summary of strengths: 1. Topic of work (inconsistencies in bias measurement) is important.
2. Good effort in controlling for semantic consistency with style transforms when the two are often linked.
Summary of weaknesses: No major complaints. It may be helpful to see human evaluations for possible semantic change for punctuation substitution and evidential markers.
Comments, suggestions and typos: None
Overall assessment scores: 4