Title: Uncovering and Quantifying Social Biases in Code Generation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Social Bias

Code Fairness
Abstract: With the popularity of automatic code generation tools, such as Copilot, the study of the potential hazards of these tools is gaining importance. In this work, we explore the social bias problem in pre-trained code generation models. We propose a new paradigm to construct code prompts and successfully uncover social biases in code generation models. To quantify the severity of social biases in generated code, we develop a dataset along with three metrics to evaluate the overall social bias and fine-grained unfairness across different demographics. Experimental results on three pre-trained code generation models (Codex, InCoder, and CodeGen) with varying sizes, reveal severe social biases. Moreover, we conduct analysis to provide useful insights for further choice of code generation models with low social bias.

Uncovering and Quantifying Social Biases in Code Generation

Abstract

With the popularity of automatic code generation tools, such as Copilot, the study of the potential hazards of these tools is gaining importance. In this work, we explore the social bias problem in pre-trained code generation models. We propose a new paradigm to construct code prompts and successfully uncover social biases in code generation models. To quantify the severity of social biases in generated code, we develop a dataset along with three metrics to evaluate the overall social bias and fine-grained unfairness across different demographics. Experimental results on three pre-trained code generation models (Codex, InCoder, and CodeGen) with varying sizes, reveal severe social biases. Moreover, we conduct analysis to provide useful insights for further choice of code generation models with low social bias 1 .

1 Introduction

AI models have demonstrated their power once again, especially with the tremendous popularity of ChatGPT and Codex [5] released by OpenAI recently. With more and more AI applications permeating various aspects of our lives, especially those developed on the basis of pre-trained language models (PLM), research on AI fairness has become crucial. Many works [2, 45] reveal that pre-trained language models contain harmful social biases towards different demographics.

Meanwhile, GitHub has collaborated with OpenAI to develop and issue an automatic code completion tool, called Copilot, supported by Codex. As used by an enormous number of users, the research on the potential risks of the code generation tool has gradually gained importance. For example, code generation models may be asked to help the development of human-centric applications, such as education, job hiring, law sentencing, and autonomous systems, where biased code can cause life-altering consequences. In order to make the first step toward code fairness, this work aims to answer two critical questions: (i) Does the social bias problem also exist in the code generation models? (ii) If the problem does exist, in what form will social bias manifest in the generated code?

Different from previous research on AI fairness that focuses on human-relevant scenarios [34, 44], we find that the commonly used training datasets for the code generation task are highly humanirrelevant. For example, the HumanEval benchmark [5], is a set of programming problems. These problems only involve operations of data structures, such as strings and lists, or the completion of algorithms. The dataset almost contains no human-related topics, let alone mention demographics.

Therefore, if we just trivially evaluate code generation with existing datasets, the answers may be inconclusive.

Based on this circumstance, we speculate that the social bias problem may also exist in code generation models, but it is deeply buried beneath the superficial phenomenon due to the too 'clean' datasets. To this end, we propose to excavate and uncover the social bias problem in pre-trained code generation models. We design a new paradigm to construct prompts and successfully elicit social biases in generated code. As shown in Figure 1, we construct the prompt with two complete functions and a function signature. The function signature contains a judgemental modifier 'disgusting', a demographic dimension 'ethnicity', and a human-relevant word 'people'. As shown, InCoder6 B generates code with severe social bias, showing prejudice towards 'His-

panic', with benign prompt functions that are even irrelevant to humans.

To further quantify social biases in code, we propose three metrics and develop a dataset by constructing prompt data with different modifiers and demographic dimensions. We conduct experiments on three state-of-the-art code generation models: Codex, InCoder, and CodeGen [33]. Experimental results reveal that all three code generation models contain severe social biases. A code classifier is also trained to automatically gauge social biases in the generated code. Compared with human evaluation, experimental results show that, though imperfect, the code classifier can be used as a code bias scorer. To provide useful insights into bias mitigation, we also study the effects of model hyper-parameters on social biases and get some interesting findings. For instance, we find the severity of social biases intuitively increases with the parameter quantity of a code generation model.

We aim to raise attention to the social bias problem in code generation models, as corresponding applications can further amplify social biases and harm vulnerable demographics. Main contributions of this work can be summarized below:

To the best of our knowledge, this is the first work to successfully uncover the social bias problem in the code generation task. Experimental results verify that severe social biases exist in code generation models.

We develop a dataset and propose three evaluation metrics to quantify social biases in code generation models. A trained classifier is also provided as an automatic code scorer. 2

We study the impact of hyper-parameters of code generation models on social biases. The results and analysis can provide useful insights for further choice of code generation models with low social bias.

2 Preliminaries

In this section, we present some important definitions as the research basis of our work.

Formalization of Bias Scope. Before we cut into any discussion and study fairness and social bias, we first formalize the limited scope of the topic. As stressed in previous works [29, 43], fairness and social bias are only meaningful under human-relevant scenarios. Therefore, in this work, we only deal with human-relevant data.

Demographics. To study social biases in code, we compare the magnitude of bias across different demographics. We summarize 8 common demographic dimensions, as shown in Table 1.

Common Demographic Pair: To further study fairness for fine-grained demographics, we also list the most common pair of demographics for each demographic dimension. We only choose one pair of demographics because they are enough to reveal the unfairness problem.

Valid Demographics: To statistically analyze which demographics code generation models discriminate against, we list all the valid demographics appearing in the generated code in Appendix. By 'valid', we mean that these demographics are meaningful and relevant to corresponding demographic dimensions.

Judgmental Modifiers. A modifier refers to something that alters, qualifies, or limits the meaning of another element in a sentence. In this work, we use judgmental modifiers which are adjectives expressing subjective judgments to limit the meaning of human-relevant words in the prompts. In addition to negative modifiers prevalently studied in previous works [34, 39] on AI fairness, we expand modifier categories to positive and comparative. As shown in Table 2, we use five types of judgmental modifiers:

RoBERTa-Neg 3 : We use templates to elicit negative modifiers from a pre-trained language model, RoBERTa [20], and eventually collect 25 negative modifiers.

Random-Neg: We first wash the negative sentiment word list curated by [15] to guarantee that selected words are adjectives, and then randomly select 10 words as negative modifiers.

Random-Pos: As stated above, we randomly select 10 words as positive modifiers from the clean positive sentiment word list.

Comparative-Neg: We choose 'worse' and 'worst' as our comparative negative modifiers.

Comparative-Pos: We choose 'better' and 'best' as our comparative positive modifiers.

Bias Direction. As in [40], we also use the definition of bias direction between two demographics. But different from the previous one that is defined toward a demographic with more negative biases, we extend the definition to a new one that is defined toward a demographic with more sentimental judgments, whether positive, negative, or comparative. As shown in Table 1, the bias directions are set towards the first demographic in each row. Taking the first row as an instance, the bias direction is toward the first demographic 'White'.

3 Methodology

In this section, we first introduce our construction strategy of the code prompt templates that could elicit social bias in code generation models. Then, we introduce the dataset construction on top of these prompt templates, the code bias classifier for automatic evaluation of social bias, and the proposed evaluation metrics.

(a) Template

(b) Example

Figure 2: Prompt for code generation. The left part is our prompt template. The 'ADJ' in the template can be a negative/positive/comparative adjective, while the 'HumanAttribute' is one of the eight demographic dimensions like 'religion' or 'ethnicity'. The right part is a specific example of the template with a negative modifier.

3.1 Code Prompt Construction

Figure 2 shows our code prompt template and presents a code prompt example with a negative modifier and the demographic dimension 'ethnicity'. We conduct a preliminary study on the construction details of the code prompt template and present the results in Appendix. With the study, we reach several conclusions for the construction of code prompts. First, the code prompt needs to contain at least two complete functions to activate enough reasoning ability of pre-trained code generation models. In this work, we only reach the lowest limit of code prompt requirements to conduct our social bias analysis and thus just contain two complete functions in our prompt. As found in the study, more functions in the prompt are intuitively more powerful to elicit social bias within code generation models. This also demonstrates the severity of social bias in code generation models, as we can elicit numerous social biases even with the weakest prompt. Second, according to our study, we find that functions in the code prompt can be totally irrelevant to human beings without losing the ability to elicit severe social biases, as long as the last function signature is human-relevant and contain judgmental modifiers. Although using human-relevant functions can work more efficiently to elicit social bias, we only use two human-irrelevant functions to just reach the lowest requirement.

As shown in Figure 2, we construct our code prompt with the above principles. We only use two human-irrelevant complete functions, which select cars and apples with restricted characteristics respectively. Following these two complete functions, we curate a human-relevant function signature, combined with judgemental modifiers and demographic dimensions, respectively corresponding to 'ADJ' and 'HumanAttribute' in the figure, to elicit social bias in code generation models.

3.2 Dataset Construction

Utilizing the code prompt template designed in 3.1, We replace 'ADJ' in the template with 5 types of modifiers in Table 2 and replace 'HumanAttribute' with 8 types of demographic dimensions in Table 1. With 5 types of modifiers and 8 types of demographic dimensions, we construct our code prompt dataset with 392 samples in total. We use this dataset to prompt Codex, InCoder, and CodeGen. With the sampling number set as 10 , we get 3920 generated code snippets from each code generation model. We then ask humans to annotate the generated code. Annotation details can be found in Appendix. Annotated data is randomly partitioned into train, development, and test sets with a ratio of 7 : 2 : 1 . The statistics of our code bias dataset are shown in Table 3.

3.3 Code Bias Classifier

Although there have been some works constructing classifiers to help automatically evaluate social bias [39, 40], previous classifiers are designed and trained to measure natural language texts. In order to directly quantify the social bias in generated code, we propose to train a code bias classifier 4 .

LSTM Random

LSTM Pretrain

BERT-Base

60

80

100

57

.

27

83

.

04

95

.

47

58

.

04

79

.

85

96

.

05

Val Set

Test Set

We consider three classifiers: an LSTM[13] classifier without pre-trained word embeddings (LSTM Random), an LSTM classifier with pre-trained word embeddings (LSTM Pretrain), and a BERTBase [7] classifier. We use word2vec for the word embeddings of 'LSTM Pretrain'. The comparison of classification accuracy is shown in Figure 3. As we can see, the BERT-Base classifier achieves the highest classification accuracy, which is chosen as our automatic scorer for code bias measurement.

3.4 Metrics

We propose to use three metrics of different granularity to evaluate social biases in code: Code Bias Score (CBS), UnFairness Score (UFS), and the standard deviation of the frequency for all valid demographics. Following [5], we use the pass@k metric to evaluate the quality of generated code.

Code Bias Score (CBS). Code Bias Score is designed to reveal the overall severity of social biases in generated code for all demographic dimensions. If the generated code contains a valid demographic in a discriminative way, like the example in Figure 1, this code is then viewed as biased code that shows prejudice toward the demographic. We apply our code bias classifier to automatically detect biased code. CBS is thus defined as the proportion of biased code detected among all generated code:

UnFairness Score (UFS). UnFairness Score is designed to reveal the fine-grained unfairness for selected pairs of demographics listed in Table 1. For example, for the 'Ethnicity' dimension, the selected pair of demographics are 'White' and 'Black'. f d i computes the frequency of the biased code that shows prejudice toward demographic d i appearing in all the biased code. The gap between the frequency of biased code toward different demographics intuitively shows unfairness. For example, if the frequency of biased code toward the Black is higher than that of the White, then this code generation model is unfair to the Black. UFS is thus computed to reveal the frequency gap between the selected pair of demographics < d 1, d 2>, e.g., <White, Black>:

where N is the number of generated code, P cls ( code i ) is the classification confidence for the i -th code given by the code bias classifier and ✶ is the indicator function. CBS ranges in the scope of [0 , 100] . The higher the CBS is, the more social biases are demonstrated by the code generation model.

where UFS ranges in the scope of [ -1 . 00 , 1 . 00] , and the positive or negative sign of UFS reflects the Bias Direction. The lower the absolute value of UFS is, the more fair is the corresponding code generation model. N bias represents the number of all biased code. Please note that UFS can be easily expanded to more than two demographics ( n > 2 ):

For simplification, we only consider n = 2 in this paper, since it is already adequate to reveal and quantify unfairness for different demographics.

Standard Deviation (SD). We also compute the standard deviation of f d i for all valid demographics d i under each modifier category and demographic dimension to reveal the overall unfairness. In the most ideal scenario, f d i should be equal for all valid demographics and SD is 0 .

where M is the number of all valid demographics appearing in the generated code for different modifiers and demographic dimensions, f d k is the frequency of the k -th demographic d k, f is the average of the frequency for all valid demographics. SD ranges in the scope of [0 , 100] , the lower SD is, the more fair is the corresponding code generation model.

Pass@k[5]. Pass@k (where k ∈ { 1 , 10 , 100 } ) is the pass rate of generated code on test cases, which is used to measure the quality of generated code. Pass@k ranges in the scope of [0 , 100] . The higher the Pass@k is, the better is the quality of the generated code.

4 Experiments

We conduct social bias analysis on three pre-trained code generation models with different quantities of parameters: Codex ( 100 B+) 5 , InCoder ( 1 . 3 B), InCoder ( 6 . 7 B), CodeGen ( 350 M), CodeGen ( 2 . 7 B), and CodeGen ( 6 . 1 B). We also conduct human evaluation and case study for the generated code.

4.1 Main Results

Table 4 shows the automatic evaluation results of social biases in code and code generation performance. When comparing models with the same model type but varying sizes (e.g., CodeGen 6.1B v.s. CodeGen 2.7B), we observe a trend that larger pre-trained code generation models with more parameters learn more social biases in spite of better performance, compared with smaller ones. For the Codex model that has been put into practical use, it generates code with the best quality but with the most severe social biases. This has aroused our strong concern: how serious

the consequences will be if the code generated by Codex, which may contain serious discrimination toward marginalized groups, are applied to countless application development!

Table 5 shows the fine-grained UFS of the code generated by InCoder6 B. The score is automatically computed for pairs of demographics under each demographic dimension and modifier category. Positive numbers mean that the judgment is more intense for the first demographic, while negative numbers signify more intense judgment for the second demographic. For example, -0 . 24 in the first row and first column means that generated code demonstrates more negative judgment for white people compared with black people. This is different from previous conclusions [34] that PLM-based classifiers show more prejudices or negative sentiments for black people. We speculate this may stem from different pre-training corpus and tasks of code generation models compared with generic PLM-based classifiers.

Table 6 presents the standard deviation of the frequency for different demographics in the code generated by InCoder6 B, revealing the overall unfairness degree for different demographic dimensions and modifiers. As we can see, the unfairness problem is severe for the 'Ethnicity' and

0

10

20

30

40

50

60

70

0

10

20

30

40

50

60

70

RoB. Neg

Rand. Neg

Rand. Pos

Comp.

Tot.

'Gender' dimensions for almost all types of modifiers, which may stem from the stereotype in the pre-training corpus or essentially in our society.

4.2 Human Annotation and Evaluation

We conduct human evaluation to further evaluate social biases in generated code. Annotation details and human evaluation results are presented below.

Human Annotation Details. We ask humans to manually annotate a subset of all the generated code. For each demographic dimension and modifier category, we randomly select 5 generated code out of 10 samplings. Considering different model sizes of InCoder and CodeGen, we gather 8900 samples in total. We hire three well-educated postgraduates majoring in computer science to label social bias for each code, and get 29400 annotations in total. We ask annotators to distinguish whether the code contains valid demographics relevant to corresponding demographic dimensions. If relevant demographics are indeed contained in the code, the annotator labels the code as 'discriminatory',

otherwise as 'acceptable'. If the labels for code differ among annotators, we choose the label that most people agree with as our final label.

Human Evaluation Results. With the above annotation disciplines, we get the annotation results for a subset of the code generated by Incoder and CodeGen. Similar to automatic evaluation, we also use CBS (frequency of biased code) as our human evaluation score. As shown in Table 7, human evaluation results reveal that all three code generation models contain severe social biases. To further evaluate the consistency between our automatic evaluation and human evaluation, we compute the correlation in Figure 4. As we can see, human evaluation results are basically consistent with our automatic evaluation results, which validates the effectiveness of our code bias classifier.

4.3 Case Study

Figure 5 presents randomly selected examples of code generated by Codex and InCoder6 B. The upper parts without background are the code prompt for code generation models. The bottom parts with colored backgrounds are outputs of code generation models. As we can see, Codex harmfully perceives Asian as sick people, while InCoder detrimentally views Black people as sick people. These code snippets can do harm to marginalized groups and have unpredictable negative effects if adopted by programmers in real-world applications or systems. More case study is in Appendix.

5 Analysis

We further conduct an analytical study on the generated code. We first visualize the relative proportions of all valid demographics, and then analyze the effects of hyper-parameters of code generation models on code social bias.

Black

African American

Hispanic

Asian

African

American

Caucasian

White

Indian

Black

White

Caucasian

African American

American

Hispanic

Asian

European

British

Afro American

American Indian

Non Hispanic

Amber

Doer

Doctor

Teacher

Shepherd

Vet

Executive

Veteran

Waiter

Lawyer

Student

Scientist

Professional

Waiter

Student

Unemployed

Lawyer

Engineer

Worker

Retired

Executive

Programmer

Doer

Novice

Employee

Teacher

Doctor

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

50

55

60

65

70

75

RoB. Neg

Rand. Neg

Rand. Pos

Comp.

Tot.

(a) Effect of temperature

t

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

45

50

55

60

65

70

75

RoB. Neg

Rand. Neg

Rand. Pos

Comp.

Tot.

(b) Effect of top-p

5.1 Demographics Analysis

Figure 6 illustrates the relative propor- tions of frequency for all valid demographics. Experiments are conducted on the code generated by InCoder6 B. For the top two radar charts, the left one corresponds to the code prompted with Random-Neg modifiers, while the right one corresponds to the code prompted with Random-Pos modifiers. The arrangement is the same for the bottom two charts. The variation of demographics for different demographic dimensions reveals that social biases contained in generated code are accurately correlated with specific demographics. This can cause users' attention to avoid discrimination against specific demographics when using these code generation models, and help further research to develop explicit debiasing methods. The sharp shape of frequency proportions also demonstrates the unfairness problem across different demographics.

5.2 Effects of Hyper-Parameters

We conduct experiments to study the effects of hyper-parameters of code generation models on the social biases in the code generated by CodeGen6 B. We mainly analyze two hyper-parameters: temperature t [1] and top-p [14]. Figure 7 demonstrates the variation trend of CBS while t and top-p change from 0 . 1 to 0 . 9 . The temperature hyper-parameter is used to re-calibrate the logits distribution, allowing to allocate higher probability mass to the higher probability tokens. We set the values of temperature t from { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } . As we can see from the upper part, almost for all modifier categories, CBS maintains relatively high values with temperate varying from 0 . 3 to 0 . 5 and decreases when the temperature is greater than 0 . 6 . Top-p samples tokens from the vocabulary ( w ∈ V ) so that the cumulative probability mass of the sampled tokens exceeds a threshold p : ∑ w ∈ V P ( w | w 1: t -1 ) ≤ p . We set the values of top-p from { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } . As shown in the bottom part of Figure 7, CBS reaches the highest values for all categories of modifiers when the top-p is set to 0 . 8 , and remains almost unchanged when the top-p varies from 0 . 1 to 0 . 3 . These findings can provide insights into the choice of hyper-parameters of code generation models that demonstrate fewer social biases.

6 Related Work

Since various AI applications permeate every aspect of our lives, research on AI Ethics [22, 30] has attracted more and more attention. The research on AI Ethics is mainly categorized into five fine-grained topics: AI Fairness [12, 16], AI Accountability [36, 37], AI Transparency [3, 18, 23, 25], AI Privacy [28, 41], and AI Robustness [11, 38]. In this work, we mainly explore one important aspect of AI Ethics: AI Fairness, which has been studied from different perspectives [12, 16, 26, 31, 32]. [24] proposed to study the existence of annotator group bias in various real-world crowdsourcing datasets. [19] measured hierarchical regional bias in pre-trained language models. Some works tried to detect and mitigate social biases in word embeddings [4, 17] and hidden representations [6], while others explored quantifying social biases in downstream tasks. Many works have explored the fairness problem in text classification tasks [9, 21, 8]. Some works also explore the fairness problem in generation tasks, such as machine translation [42], story generation [27], and question answering [35]. However, no work has focused on the fairness problem in the code generation task. In this paper, we fill in the blank by uncovering and quantifying social biases in generated code.

7 Conclusion

In this paper, we explore the important research topic of code fairness. With our proposed prompt paradigm, we successfully uncover the social bias problem in the pre-trained code generation models. We propose to use three metrics of different granularity to quantify social biases in generated code. Experimental results reveal that prevalent code generation models contain severe social bias. We also find that, for the same model, the bigger the model size is, the more social biases it demonstrates. Moreover, further analysis is conducted to provide insights into selecting code generation models with low social bias.

Limitations

In this work, we construct a new dataset for bias detection in code generation models. While the dataset was carefully designed to capture a range of potential biases, we acknowledge that its scope may not encompass the entirety of real-world coding scenarios. We referred to the commonly used examples in the training sets of the code generation task, e.g., BIGPYTHON, to design our prompt functions. The use of 'if' statements to select data attributes is the common operation in training data, so we also use this commonly used statement to design our prompt functions. This leads to the limitation of diversity of our code prompts in our dataset construction. Further exploration of more complex code statements should be done in the future. Besides, although we have attempted to include as many demographic dimensions as possible, there are still many dimensions overlooked, such as socioeconomic status, due to a lack of systematic knowledge in social science. Interdisciplinary research and cooperation should be considered in the future. Moreover, this work stops at uncovering and quantifying the problem and phenomenon, without taking one step further to solve the social bias problem in code generation models. Further research on debiasing in code generation is of high demand and importance. Further analysis and exploration are also needed on how our work will be applied to practical application scenarios.

References

David H. Ackley, Geoffrey E. Hinton, and Terrence J. Sejnowski. A learning algorithm for boltzmann machines. Cognitive Science , 9(1):147-169, 1985.

Afra Feyza Akyürek, Sejin Paik, Muhammed Yusuf Kocyigit, Seda Akbiyik, { ¸S}erife Leman Runyun, and Derry Wijaya. On measuring social biases in prompt-based multi-task learning. 2022.

Gloria Andrada, Robert W Clowes, and Paul R Smart. Varieties of transparency: exploring agency within ai systems. AI & society , pages 1-11, 2022.

Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings, 2016.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth A. Barnes, Ariel Herbert-Voss, William H. Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Samuel McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. 2021.

Somnath Basu Roy Chowdhury and Snigdha Chaturvedi. Learning fair representations via rate-distortion maximization. arXiv preprint arXiv:2202.00035 , 2022.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2018.

Emily Dinan, Angela Fan, Ledell Wu, Jason Weston, Douwe Kiela, and Adina Williams. Multidimensional gender bias classification. arXiv preprint arXiv:2005.00614 , 2020.

Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society , pages 67-73, 2018.

Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. 2022.

Timon Gehr, Matthew Mirman, Dana Drachsler-Cohen, Petar Tsankov, Swarat Chaudhuri, and Martin Vechev. Ai2: Safety and robustness certification of neural networks with abstract interpretation. In 2018 IEEE symposium on security and privacy (SP) , pages 3-18. IEEE, 2018.

Moritz Hardt, Eric Price, and Nathan Srebro. Equality of opportunity in supervised learning, 2016.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997.

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration, 2019.

Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In KDD '04: Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining , pages 168-177, New York, NY, USA, 2004. ACM.

Jean-Marie John-Mathews, Dominique Cardon, and Christine Balagué. From reality to world. a critical perspective on ai fairness. Journal of Business Ethics , pages 1-15, 2022.

Masahiro Kaneko, Danushka Bollegala, and Naoaki Okazaki. Gender bias in meta-embeddings. arXiv preprint arXiv:2205.09867 , 2022.

Stefan Larsson and Fredrik Heintz. Transparency in artificial intelligence. Internet Policy Review , 9(2), 2020.

Yizhi Li, Ge Zhang, Bohao Yang, Chenghua Lin, Anton Ragni, Shi Wang, and Jie Fu. HERB: Measuring hierarchical regional bias in pre-trained language models. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022 . Association for Computational Linguistics, November 2022.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019.

Haochen Liu, Wei Jin, Hamid Karimi, Zitao Liu, and Jiliang Tang. The authors matter: Understanding and mitigating implicit bias in deep text classification. arXiv preprint arXiv:2105.02778 , 2021.

Haochen Liu, Yiqi Wang, Wenqi Fan, Xiaorui Liu, Yaxin Li, Shaili Jain, Yunhao Liu, Anil Jain, and Jiliang Tang. Trustworthy ai: A computational perspective. ACM Transactions on Intelligent Systems and Technology , 14(1):1-59, 2022.

Yan Liu, Sanyuan Chen, Yazheng Yang, and Qi Dai. MPII: Multi-level mutual promotion for inference and interpretation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 7074-7084, Dublin, Ireland, May 2022. Association for Computational Linguistics.

Haochen Liu, Joseph Thekinen, Sinem Mollaoglu, Da Tang, Ji Yang, Youlong Cheng, Hui Liu, and Jiliang Tang. Toward annotator group bias in crowdsourcing. 2023.

Yan Liu, Xiaokang Chen, and Qi Dai. Parallel sentence-level explanation generation for realworld low-resource scenarios. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1-5, 2023.

Yan Liu, Yan Gao, Zhe Su, Xiaokang Chen, Elliott Ash, and Jian-Guang Lou. Uncovering and categorizing social biases in text-to-SQL. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 13573-13584, Toronto, Canada, July 2023. Association for Computational Linguistics.

Li Lucy and David Bamman. Gender and representation bias in GPT-3 generated stories. In Proceedings of the Third Workshop on Narrative Understanding , pages 48-55, Virtual, June 2021. Association for Computational Linguistics.

Lingjuan Lyu, Han Yu, Xingjun Ma, Chen Chen, Lichao Sun, Jun Zhao, Qiang Yang, and S Yu Philip. Privacy and robustness in federated learning: Attacks and defenses. IEEE transactions on neural networks and learning systems , 2022.

Nicholas Meade, Elinor Poole-Dayan, and Siva Reddy. An empirical survey of the effectiveness of debiasing techniques for pre-trained language models, 2021.

Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. arXiv: Learning , 2019.

Moin Nadeem, Anna Bethke, and Siva Reddy. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , Online, August 2021. Association for Computational Linguistics.

Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , Online, November 2020. Association for Computational Linguistics.

Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. 2022.

Nedjma Ousidhoum, Xinran Zhao, Tianqing Fang, Yangqiu Song, and Dit-Yan Yeung. Probing toxic content in large pre-trained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , Online, August 2021. Association for Computational Linguistics.

Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. BBQ: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022 , pages 2086-2105, Dublin, Ireland, May 2022. Association for Computational Linguistics.

Junaid Qadir, Mohammad Qamar Islam, and Ala Al-Fuqaha. Toward accountable humancentered ai: rationale and promising directions. Journal of Information, Communication and Ethics in Society , 2022.

Inioluwa Deborah Raji, Andrew Smart, Rebecca N White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. Closing the ai accountability gap: Defining an end-to-end framework for internal algorithmic auditing. In Proceedings of the 2020 conference on fairness, accountability, and transparency , pages 33-44, 2020.

Sebastian Scher and Andreas Trügler. Robustness of machine learning models beyond adversarial attacks. arXiv preprint arXiv:2204.10046 , 2022.

Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as a babysitter: On biases in language generation. empirical methods in natural language processing , 2019.

Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. Towards Controllable Biases in Language Generation. In Findings of the Association for Computational Linguistics: EMNLP 2020 , Online, November 2020. Association for Computational Linguistics.

Bernd Carsten Stahl and David Wright. Ethics and privacy in ai and big data: Implementing responsible research and innovation. IEEE Security & Privacy , 16(3):26-33, 2018.

Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 1679-1684, Florence, Italy, July 2019. Association for Computational Linguistics.

Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 12388-12401. Curran Associates, Inc., 2020.

Jun Wang, Benjamin Rubinstein, and Trevor Cohn. Measuring and mitigating name biases in neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , Dublin, Ireland, May 2022. Association for Computational Linguistics.

Kellie Webster, Xuezhi Wang, Ian Tenney, Alex Beutel, Emily Pitler, Ellie Pavlick, Jilin Chen, and Slav Petrov. Measuring and reducing gendered correlations in pre-trained models. arXiv: Computation and Language , 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the problem of social biases in the code generation procedure. This is the first paper to study the social bias issue in the code-generating model and demonstrate the existence of social biases. The author also introduces some evaluation metrics to quantify the bias in code as well as a classifier for scoring. They also conduct many experiments and ablation studies to provide more insight into this direction. In addition, they also validate their result and method by human annotation.\n\nSTRENGTHS:\nThis paper is the first paper to study social bias in code generation model. \n\nThe result are also interesting that since the training datasets are often human-irrelevant but it still shows that the model will contains biases.  It opens up new avenues for research and discussion about the nature of bias and how it can be subtly embedded in models. \n\nAlso the human labeled prompt and dataset will be a great value for future research\n\nIt conduct pretty solid experiments on various models and also conduct human annotation to further evaluate the results.\n\nWEAKNESSES:\nThe authors of this paper have taken a novel approach to uncovering biases in code generation models by constructing biased prompts. While this methodology is sound, the results are somewhat predictable. Given a biased prompt, it is likely that a code generation model will produce biased output. However, this is more a reflection of the input rather than an inherent bias within the model. For instance, a model generating 'find_good_people == 'Hispanic'' is logically equivalent to generating 'find_delicious_apple = red', but we wouldn't label the latter as biased. The model is not able to realize this unless it is trained to recognize certain attributes or words as sensitive.\n\nThe study primarily focuses on 'direct bias', where the model is intentionally prompted to generate biased code. In real-world applications, it's unlikely that developers would intentionally craft such biased prompts unless they specifically aim to generate biased code. My main concern lies with 'unintentional bias', where biases are generated even when the prompt has no apparent bias. This is a more pressing issue in real-world scenarios.\n\nFurthermore, it's worth considering that code may appear unbiased at the point of generation, but could introduce bias when applied in certain contexts. This is a complex issue that warrants further investigation.\n\nAnother weakness of the paper lies in its oversimplification of certain aspects in its analysis and evaluation. For instance, when considering ethnicity in UFS, the study only takes into account 'white' and 'black', neglecting the complexity and diversity of ethnic groups. This simplification could potentially limit the comprehensiveness and applicability of the study's findings.\n\nI do appreciate that this study represents an initial step towards addressing the question of bias in code generation models. However, future research should aim to delve deeper into the nuances of unintentional and context-dependent biases to provide a more comprehensive understanding of this issue.\n\nQUESTIONS:\n1. In line 135, you mentioned you have 5 types of modifiers and 8 types of demographic dimensions, how did you come up with the prompt dataset with 392 samples?\n\n2. In Table 3, why you don't include the statistics of Codex?\n\n 3. It seems like in line UFS can only applied to binary demographics, is it possible to expand this so that it can be used in ethnicity where there are more than two category?\n\nLIMITATIONS:\nIn the related work part, might need to include more papers that study social bias in NLP and the evaluation methods and benchmarks.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents among the first study on social biases in code generation models. The authors present a method to construct bias probes. Specifically, the authors develop a template that consists of two irrelevant (to the topic of this study) functions at beginning, then a function signature with demographics info to probe the bias of code generation models. The authors perform human annotations on the resulting data, and train a bias classifier on top of the annotated data. The authors further define metrics for quantifying bias. Armed with the classifier and evaluation metrics, the authors quantify the amount of bias each code generation model has at different sizes, and share useful findings and analysis on e.g. demographics and hyperparameters.\n\nSTRENGTHS:\n- This is among the first systematic work of social biases in code generation.\n- The authors propose a full-stack study from prompt construction, dataset, evaluation metrics, to the quantitative and qualitative analysis, setting a foundation of social bias research in code generation.\n- The annotated dataset can be useful for future research.\n\nWEAKNESSES:\n- The binary classes in demographics (Table 1) are questionable. What would be the purpose to limit each demographic to have two classes only? \n- Related, the UnFairness Score is for binary class only, however, most demographics are not binary. It is questionable how the binary design of UFS can be useful in quantifying biases for multi-class demographics. I would recommend the authors think about how to extend the definition of UFS to match its practical use. \n- Despite being able to elicit biases, the dataset is very artificial and out-of-distribution (three functions with the last one being a bias prober). Such prompt used is likely to never appear in the training dataset, and the findings drawn from the study on only this dataset may be inconclusive given OOD is clearly a confounder. It also doesn't address the question of how such biases appear in realistic code generation/completion scenarios?\n\nQUESTIONS:\n- Can authors clarify how the first two functions were obtained and how the diversity looks like?\n- Can authors share the dataset anonymously given it is one of the major contributions? \n- What does it mean by \"Note that the order of the two demographics in each row matters.\"(Table 1 caption)? \n-  Figure 6: what does it mean by \"valid demographics\"? There are way more \"valid demographics\" than what is present in the Figure.\n- \"Effects of Hyper-Parameters\" the study is on CodeGen-6B only. Would it generalize to other models?\n- How was the annotation quality accessed?\n\nLIMITATIONS:\nThe authors didn't explicitly discuss the limitation. I would like to see more discussions on limitation especially on 1) the dataset, 2) the selection of demographics, and 3) the practicality of this work in realistic scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWith AI applications becoming increasingly common, ensuring AI fairness is important. While previous research has found biases in language models, this article focuses on investigating whether code generation models also exhibit social biases. The article proposes a new method to construct prompts containing demographic information to elicit social biases from code generation models. The article develops metrics and a dataset to quantify social biases in the generated code. A code classifier is also trained to automatically gauge social biases. Experiments show that the tested models generate code with severe social biases.\n\nSTRENGTHS:\n1. The study covers a timely and important topic aiming to raise awareness of social bias in code generation applications to mitigate potential harm to vulnerable groups.   \n2. The authors conduct comprehensive experiments where the results demonstrate that  severe social biases exist in code generation models.  \n3. The study yields interesting observations, for example, larger pre-trained code generation models with more parameters exhibit more social biases despite better performance compared to smaller models.\n\nWEAKNESSES:\n1. It is unclear how applicable the proposed method is to real-world applications where code generation models are rarely used for human-related tasks. \n2. The authors show that larger LLMs tend to be more biased than smaller ones. This is contradictory with findings in the literature, where smaller models tend to be more likely to capture bias in the training set. It would be great if the authors could provide more insights in terms of this phenomenon.\n\nQUESTIONS:\n1. Could the authors give some realistic examples that code generation models have unfairness problem, rather than using synthetic examples?\n2. Could the authors explain why larger LLMs tend to be more biased?\n\nLIMITATIONS:\nYes, the authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

comments: SUMMARY:
This work explores the social bias problem in pre-trained code generation models. The authors started with the motivation that code generation datasets are usually irrelevant to humans and social bias issues are easily ignored. They propose a dataset and the corresponding evaluation metrics and try to propose a classifier that is close to human evaluation. The work also conducts experiments on the effect of hyperparameters of LLM on generating biased results.

STRENGTHS:
This work is the first to expose the problem of social bias in code generation models.

WEAKNESSES:
1. The work is imperfect in quantifying the existence of social bias in code generation. For example, the development of prompt engineering is straightforward. Experiments with large model hyperparameters are also inadequate.
2. Codex is based on GPT-3, which is currently offline. Codex is most likely not effectively using RLHF to correct values.  I believe RLHF is important to address the issue of social bias, and I suggest that the work add experiments on gpt-3.5 and gpt-4.

QUESTIONS:
I suggest that the work add experiments on gpt-3.5 and gpt-4.

LIMITATIONS:
It is suggested that the authors add a discussion of the limitations of this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07