Uncovering and Categorizing Social Biases in Text-to-SQL
Year: 2023

Abstract:
Content Warning: This work contains examples that potentially implicate stereotypes, associations, and other harms that could be offensive to individuals in certain social groups. Large pre-trained language models are acknowledged to carry social biases towards different demographics, which can further amplify existing stereotypes in our society and cause even more harm. Text-to-SQL is an important task, models of which are mainly adopted by authoritative institutions, where unfair decisions may lead to catastrophic consequences. However, existing Text-to-SQL models are trained on clean, neutral datasets, such as Spider and WikiSQL. This, to some extent, cover up social bias in models under ideal conditions, which nevertheless may emerge in real application scenarios. In this work, we aim to uncover and categorize social biases in Text-to-SQL models. We summarize the categories of social biases that may occur in structured data for Text-toSQL models. We build test benchmarks and reveal that models with similar task accuracy can contain social biases at very different rates. We show how to take advantage of our methodology to uncover and assess social biases in the downstream Text-to-SQL task1.

heading: 1 Introduction
text: Automated systems are increasingly being used for numerous real-world applications (Basu Roy Chowdhury et al., 2021), such as filtering job applications, determining credit eligibility, making hiring decisions, etc. However, there are welldocumented instances where AI model predictions have resulted in biased or even offensive decisions due to the data-driven training process. The relational database stores a vast of information and in turn support applications in vast areas (Hu and
1Our code and data are available at https://github. com/theNamek/Trustworthy-Text2SQL.
Tian, 2020). With the development of benchmark datasets, such as WikiSQL (Zhong et al., 2017) and Spider (Yu et al., 2018), many Text-to-SQL models have been proposed to map natural language utterances to executable SQL queries.
Text-to-SQL models bridge the gap between database manipulation and amateur users. In realworld applications, Text-to-SQL models are mainly applied by authoritative institutions, such as banks, schools, and governments. Such industries rely on AI-based applications to manipulate databases and further develop policies that will have profound impacts on various aspects of many people’s lives. For example, banks may use AI parsers to retrieve credit information, determining to whom they can make loans, without generating many bad debts. If there are unwanted prejudices against specific demographics in applied Text-to-SQL models, these stereotypes can be significantly amplified since their retrieval results are adopted by authoritative institutions to draft policies. Unfortunately, large pre-trained language models (PLMs) are actually acknowledged to contain social biases to-
13573
wards different demographics, and these wicked biases are observed to be inherited by downstream tasks. Some may suppose that these harmful biases could be forgotten or mitigated when fine-tuned on downstream neutral data that does not contain any toxic words, specific demographic keywords, or any judgemental expressions. However, as we observed through experiments, social biases are integrally inherited by downstream models even fine-tuned on neutral data, as in the Text-to-SQL task.
As shown in Figure 1, we notice that there are mainly two categories of social biases in the Textto-SQL task. One category of social bias is that Text-to-SQL models based on large pre-trained language models would build stereotypical correlations between judgemental expressions with different demographics. The other category of social bias is that PLM-based Text-to-SQL models tend to make wrong comparisons, such as viewing some people as worse or better than others because of their exam results, income, or even ethnicity, or religion. To better quantify social biases in Text-toSQL models, we propose a new social bias benchmark for the Text-to-SQL task, which we dub as BiaSpider. We curate BiaSpider by proposing a new paradigm to alter the Text-to-SQL dataset, Spider. For biases induced by judgmental expressions in the Text-to-SQL task, we analyze three scenarios: negative biases for demographics, positive biases for demographics, biases between different demographics under one demographic dimension.
Main contributions of this work include:
• To the best of our knowledge, we are the first to uncover the social bias problem for the Textto-SQL task. We formalize the definitions and
principles to facilitate future research of this important problem.
• We analyze and categorize different kinds of social biases in the Text-to-SQL task.
• We propose a novel prompt paradigm to uncover social biases for structured data, while previous works only focus on biases in unstructured data.
• We develop a new benchmark that can later be used for the evaluation of social biases in the Text-to-SQL models.
2 Definitions
In this section, we formalize some definitions to restrict and clarify the study scale of this work.
Formalization of Bias Scope. Before we cut into any discussion and study about fairness and social bias, we first formalize the limited scope of the topic. As stressed in previous works, fairness, and social bias is only meaningful under humanrelevant scenarios. Therefore, we only deal with human-relevant tables and queries in this work.
Demographics. To study social biases in structured data, we compare the magnitude of biases across different demographics. We summarize seven common demographic dimensions, as shown in Table 1. To further study the fairness between fine-grained demographics within one demographic dimension, we also list the most common pair of demographics used in the construction of our benchmark.
Bias Context. As stated in (Sheng et al., 2019a), biases can occur in different textual contexts. In this work, we analyze biases that occur in the sentimental judge context: those that demonstrate judgemental orientations towards specific demographics.
Judgmental Modifiers. In addition to negative modifiers prevalently studied in previous works on AI fairness (Ousidhoum et al., 2021a; Sheng et al., 2019b), we expand the modifier categories to positive and comparative, and summarize them as judgmental modifiers according to their commonality2. As shown in Table 3, we use four types of judgmental modifiers:
• RoBERTa-Neg: We use the templates provided by (Ousidhoum et al., 2021b) to elicit negative modifiers from a pre-trained language model, RoBERTa (Liu et al., 2019), and eventually collect 25 negative modifiers.
• Random-Neg: We first wash3 the negative sentiment word list curated by (Hu and Liu, 2004) to guarantee that selected words are all adjectives, and then randomly select 10 words as negative modifiers.
• Random-Pos: As stated above, we randomly select 10 words as positive modifiers from the clean positive sentiment word list.
2They are all human-relevant and essentially subjective judgments.
3We use the Stanza toolkit (https://stanfordnlp. github.io/stanza/) to annotate and filter out words.
• Comparative: We simply choose the 4 most commonly used comparative words (“worse”, “worst”, “better”, and “best”) as our comparative modifiers.
Stereotypical Correlation. We notice that in the Text-to-SQL task, one kind of common bias is that PLM-based Text-to-SQL models tend to build stereotypical correlations between sentimental judgments and certain demographics. For example, we observe that Text-to-SQL models tend to wrongly link “dangerous” to people with specific religions like “Muslim”.
Discriminative Comparison. Another common bias in the Text-to-SQL task is that Text-to-SQL models tend to view some demographics as better or worse than others due to some characteristics, such as exam grades, income, or even ethnicity.

heading: 3 Methodology
text: In this section, we first introduce our prompt construction paradigm for uncovering the social bias problem in structured data, and then introduce our social bias benchmark.

heading: 3.1 Paradigm
text: Previous works (Ousidhoum et al., 2021b) have explored the construction of prompt templates for unstructured data, while that for structured data is still under-explored. In this work, we propose a new prompt construction paradigm for uncovering the social bias problem in structured data. The whole paradigm structure is shown in Figure 2. As shown in Figure 1, social biases in the Text-to-SQL task mainly derive from stereotypical correlations between database queries and table items, such as columns. Therefore, we need to alter both queries and tables in the database. As stated in (Wang et al., 2020) and (Liu et al., 2021), we can view the database query, table information, and the linking
relationship between them as a triplet < q, t, r >, where q refers to the database query, t refers to the tabular data, and r is the relation between them. In the paradigm we proposed, we alter q and t to elicit stereotypical correlations r between them.
As shown in Figure 2, we first prompt GPT3 (Brown et al., 2020) to identify human-relevant tables. Since the research scope of this work is restricted to the human-centric scenario to facilitate our social bias study, we need to filter out tables that are irrelevant to humans. Given the power of large language models (LLM), we prompt GPT-3 to help pinpoint human-relevant tables in the database. The prompt template is shown in the first row of Table 2. Next, we prompt GPT-3 (Brown et al., 2020) to identify human-relevant queries. Finally, we prompt GPT-3 to paraphrase database queries. With the whole paradigm, we place “triggers” both in queries and tables, and eventually get our BiaSpider benchmark, which is further used to evaluate social biases in Text-to-SQL models. The following parts elaborate the prompt details.
Prompt GPT-3 to Identify Human-Relevant Tables. Since social bias only exists in humanrelevant scenarios, we first need to identify human-
relevant tables in databases. GPT-3 has demonstrated extensive power in many tasks with simple prompts. In this work, we explore to prompt the GPT-3 to help identify human-relevant tables in databases. The prompt template is shown in the first row of Table 2. We serialize a table, combining the main information and ask GPT-3 to identify whether the main object of the table is human.
Prompt GPT-3 to Identify Human-Relevant Queries. In the Spider dataset, for a humanrelevant table, there are several queries that are relevant or irrelevant to humans. Therefore, we need to further filter out queries that are irrelevant to humans. The prompt template is shown in the second row of Table 2.
Prompt GPT-3 to Paraphrase Database Queries. We also utilize GPT-3 to paraphrase database queries. As shown in Table 4, we curate patterns to alter database queries. We aim to add three types of modifiers listed in Table 3 into original queries with two different sentence structures. We feed the original database query and corresponding judgemental modifiers combined using the template shown in the third row of Table 2. We replace “ADJ” with modifiers and “QUERY” with database queries in the Spider dataset, and then ask GPT-3 to paraphrase the query by using the modifier to modify the human-relevant word. We aim to utilize GPT-3 to paraphrase neutral database queries into judgemental ones.

heading: 3.2 BiaSpider Benchmark
text: Utilizing GPT-3, we manually curate the Social Bias benchmark based on one of the mainstream Text-to-SQL dataset, Spider (Yu et al., 2018). Note that our proposed paradigm is scalable and can be applied to construct more data based on other Text-
to-SQL datasets. For each table from the original training and development set, we first serialize the table with a prompt template and utilize GPT-3 to help judge whether the main object of this table is human. For each filtered human-relevant table, we add 7 kinds of demographic dimensions into the table as extra columns. For each demographic dimension, we also correspondingly add one or more fine-grained demographics into the table as columns. The 7 demographic dimensions and corresponding demographics are shown in Table 1. We construct three versions of the benchmark dataset (BiaSpider v1, BiaSpider v2, BiaSpider v3), with an increasing number of demographics from zero to two. Statistics of all three versions of BiaSpider is shown in Table 5.

heading: 4 Experiments
text: After constructing the Text-to-SQL social bias benchmark, BiaSpider, we use this benchmark to quantitatively measure social bias in three Textto-SQL models based on different pre-trained language models.

heading: 4.1 Preliminary Experiments of Neutrality
text: To reveal the specialty of the corpus of the Text-toSQL task, we conduct preliminary experiments to show the neutrality of Text-to-SQL training data4. As shown in Table 6, scores for the toxicity and other toxic metrics of the Spider dataset are much lower than those of the pre-training corpus of BERT. The neutrality study of the social bias training corpus demonstrates that the Spider dataset almost contains no demographic items or toxic words.

heading: 4.2 Text-to-SQL Models
text: We conduct extensive experiments on three large pre-trained language models: BERT (Devlin et al., 2019) (RATSQL (Wang et al., 2020)), BART (Lewis et al., 2019) (UNISAR (Dou et al., 2022)), and T5 (Raffel et al., 2020) (PICARD (Scholak et al., 2021)). We also conduct analytical experiments on GPT-3. We list the statistics of all these models in Table 8. The statistics include the number of parameters, pre-training corpus, pre-training tasks, and model architectures.
4We use this Detoxify tool (https://github.com/ unitaryai/detoxify) to evaluate the data neutrality.
As we can see, both BART and T5 models are pretrained encoder and decoder, while BERT is only the pre-trained encoder. Except for the GPT-3, the number of parameters of other Text-to-SQL models is about the same magnitude.

heading: 4.3 Metrics
text: Bias Score. In this work, we define a new Bias Score to quantitatively measure social biases in generated SQLs. If at least one demographic dimension appears in the generated SQL without any explicit references in database queries, we view this SQL as a biased one. We notice that there are some samples that originally contain demographic dimensions. For example, there are some samples querying about age or gender information. In this case, if the generated SQL only contains corresponding demographics, we view this SQL as acceptable. We use the ratio of biased SQLs as the bias score to quantify social biases contained in Text-to-SQL models. Bias Score ranges in the scope of [0, 100]. The higher the Bias Score is, the more social biases are demonstrated by the generated SQLs.
Ori-ACC & ACC. We use the accuracy of the three Text-to-SQL models on the original Spider
dataset (Ori-ACC) as the evaluation metric for task performance. We also use the accuracy of the three Text-to-SQL models on our BiaSpider dataset (ACC) to reveal the accuracy degradation compared to that on the Spider dataset. Ori-ACC and ACC both range in the scope of [0, 100]. The higher the Ori-ACC and ACC are, the better is the performance of the model on the Text-to-SQL task.

heading: 4.4 Main Results
text: Table 7 shows the evaluation results of the three Text-to-SQL models based on different pre-trained language models. We observe that the RATSQL model which is fine-tuned on BERT demonstrates the most severe social bias with the highest Bias Score. The first three rows in every section of the table reflect stereotypical correlations with different judgemental modifiers, while the fourth row in every section presents the discriminatory comparison. Two types of social biases contained in the UNISAR and the PICARD models are about the same level revealed by the Bias Score. We can see that the Text-to-SQL models with similar task accuracy can exhibit varying degrees of social biases. Users should make a tradeoff between task performance and social biases in order to choose a more suitable model.

heading: 4.5 Case Study
text: Table 10 presents some randomly selected examples generated by different Text-to-SQL models. We notice that using the data samples generated by our proposed paradigm, all these three Textto-SQL models based on different pre-trained language models demonstrate severe stereotypical behavior. For data samples where Text-to-SQL models generate harmful SQLs, compared with ground truth SQLs, these models generate complete subclauses to infer demographic dimensions such as “Ethnicity” for the judgemental modifiers inserted before the human-relevant words in the database queries. With our proposed paradigm, we successfully elicit social biases learned by Text-to-SQL models without triggering unwanted behavior such as generating illogical SQLs.

heading: 5 Discussion
text: Q1: When should models respond to subjective judgment in queries? Like stated in (Wang et al., 2022), existing Text-to-SQL models fail to figure out what they do not know. For ambiguous questions asking about the information out of the scope of the database, current Text-to-SQL models tend to “guess” a plausible answer with some harmful grounding correlations, such as grounding “nurse” to “female”. For our case, Text-to-SQL models tend to refer to demograhic information for the judgemental modifiers, which the database has no relevant information about. We argue that no matter whether the table contains columns relevant to the judgemental modifier in the database query, Text-to-SQL models should not generate SQL that links the judgemental modifier to totally irrelevant demographic features, resulting in discriminative behaviors toward marginalized demographics. Instead, Text-to-SQL models should have the ability to figure out which restrictive information they
have no access to within the scope of the current database. This is to say, if the judgemental information, such as “is_depressed” is contained in the table, then the model would be free to infer this column. But if the database does not contain any information related to the judgemental modifier in the query, then the model should realize that it lacks information to deal with the modifier and ignore it.
Q2: What might be the reason for fewer social biases in models fine-tuned on BART and T5 than the model fine-tuned on BERT? As summarized in Table 8, we speculate that one reason for fewer social biases in models fine-tuned on BART and T5 is that these two PLMs are pre-trained encoder and decoder, while BERT is just pre-trained encoder. But whether the pre-trained decoder actually alleviates social biases for generation tasks remains to be explored in the future. Besides, the pre-training corpus for BERT may contain more toxicity than those used by BART and T5, since T5 is pre-trained on the C4 dataset, of which one “C” means “Clean”.

heading: Q3: Does different in-context learning algorithms affect social biases in generated SQL?
text: Previous works tend to attribute social biases contained in large pre-trained language models to stereotypes buried in the large pre-training corpus considering the data-driven training process. In addition to this cause, with the popularity of incontext learning in place of fine-tuning, we also wonder whether different in-context learning algorithms activate different levels of social biases. In this work, we conduct an analytical study with GPT-3.5, and explore the effects of different incontext learning algorithms. As shown in Table 9, we can see that social biases contained in the model using the DTE (Duel Transformer Encoder) and TST-Jacard (Target Similarity Tuning) (Poesia et al., 2022) algorithms is about the same, a little bit more severe than that using the TST-StringDistance (Poesia et al., 2022) algorithm. We find that this is partly due to the reason that the TSTString-Distance algorithm can accurately retrieve the most relevant sample that does not contain the judgemental modifier compared with the prompt. This makes the pre-trained language models avoid demonstrating social biases.

heading: 6 Related Work
text: The recent prosperity of AI has aroused attention in the study of AI Ethics, which mainly includes five different aspects: fairness, accountability (Liu et al., 2022, 2023), transparency, privacy, and robustness. There has been a bunch of works (Li et al., 2022) studying AI fairness in the field of Natural Language Processing(NLP). Many previous works explore to utilize template-based approach (Ousidhoum et al., 2021b; De-Arteaga et al., 2019) to detect and measure social biases in NLP models. Benchmark datasets for many tasks, such as text classification (Dixon et al., 2018), question answering (Parrish et al., 2021) for measuring social biases have already been proposed. The Text-to-SQL task is an important task, which translates natural language questions into SQL queries, with the aim of bridging the gap between complex database manipulation and amateurs. Social biases in the Text-toSQL models can cause catastrophic consequences, as these models are mainly adopted by administrative industries such as the government and banks to deal with massive data. Policies or loan decisions made by these industries based on stereotypical Text-to-SQL models can have harmful effects on the lives of innumerable people. In this work, we first verify counter-intuitively that large pre-trained language models still transfer severe social biases into “neutral” downstream tasks. For “neutral” we mean that these downstream tasks are fine-tuned on neutral corpora that are free from mentioning any demographics or judgemental expressions towards human beings. We further propose a novel paradigm to construct a social bias benchmark for the Text-to-SQL task. With this benchmark, we quantitatively measure social biases in three pretrained Text-to-SQL models.

heading: 7 Conclusion
text: In this paper, we propose to uncover and categorize social biases in the Text-to-SQL task. We propose a new paradigm to construct samples based on structured data to elicit social biases. With the constructed social bias benchmark, BiaSpider, we conduct experiments on three Text-to-SQL models that are fine-tuned on different pre-trained language models. We show that SQLs generated by stateof-the-art Text-to-SQL models demonstrate severe social biases toward different demographics, which is problematic for their application in our society by many administrative industries.
Limitations
In this work, we are the first to uncover the social bias problem in the Text-to-SQL task. We categorize different types of social biases related to various demographics. We present a new benchmark and metric for the social bias study in the Text-to-SQL task. However, this work stops at the point of uncovering and analyzing the problem and phenomenon, without making one step further to solve the social bias problem in the Text-to-SQL task. Besides, in spite of the structured scalability of our proposed paradigm for social bias benchmark construction, the efficacy of entending with other Text-to-SQL datasets remains to be verified.

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: In the section after the conclusion, without a section number.
7 A2. Did you discuss any potential risks of your work? We didn’t discuss potienal risks, because to the best of our knowledge, the research topic does not introduce additional risks.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Left blank.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 6

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Left blank.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We find it unnecessary.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? We find it unnecessary.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments? Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: But whether the pre-trained decoder actually alleviates social biases for generation tasks remains to be explored in the future. Besides, the pre-training corpus for BERT may contain more toxicity than those used by BART and T5, since T5 is pre-trained on the C4 dataset, of which one “C” means “Clean”