Uncovering and Quantifying Social Biases in Code Generation
Authors: Yan Liu
Xiaokang Chen
Yan Gao
Zhe Su
Fengji Zhang
Daoguang Zan
Jian-Guang Lou
Pin-Yu Chen
Tsung-Yi Ho
Venue: NeurIPS 2023 poster (2023)
Keywords: Social Bias
Code Fairness

Abstract: With the popularity of automatic code generation tools, such as Copilot, the study of the potential hazards of these tools is gaining importance. In this work, we explore the social bias problem in pre-trained code generation models. We propose a new paradigm to construct code prompts and successfully uncover social biases in code generation models. To quantify the severity of social biases in generated code, we develop a dataset along with three metrics to evaluate the overall social bias and fine-grained unfairness across different demographics. Experimental results on three pre-trained code generation models (Codex, InCoder, and CodeGen) with varying sizes, reveal severe social biases. Moreover, we conduct analysis to provide useful insights for further choice of code generation models with low social bias.

Uncovering and Quantifying Social Biases in Code Generation
Yan Liu
Xiaokang Chen
Yan Gao
Zhe Su
Fengji Zhang
Daoguang Zan
Jian- Jian-Guang LOU* Pin-Yu Chen
Tsung-Yi Ho
Microsoft Research Peking University
The Chinese University of Hong Kong IIBM Research
(runningme ho. tsungyi, fenj..hangg@gmail.. com,
pkucxkēpku. edu. cn, zhesu@andrew@cmuu. edu, daoguang@i cn, pin-yu. chen@ibm.. com, Cyan. gao, jlouj@microsoft. com
Abstract
With the popularity of automatic code generation tools, such as Copilot, the study of the potential hazards of these tools gaining importance. In this work, we explore the social bias problem in pre-trained code generation models. We propose new paradigm to construct code prompts and successfully uncover social biases in code generation models. To quantify the severity of social biases in generated code, we develop dataset along with three metrics evaluate the overall social bias and fine-grained unfairness across different demographics. Experimental results three pre-trained code generation models (Codex, InCoder, and CodeGen) with varying sizes, reveal severe social biases. Moreover, we conduct analysis to provide useful insights for further choice of code generation models with low social bias".
Introduction
Al models have demonstrated their power once again, especially with the tremendous popularity of ChatGPT and Codex [5] released by OpenAl recently. With more and more Al applications permeating various aspects of our lives, especially those developed on the basis of pre-trained language models (PLM), research on AI fairness has become crucial. Many works [2, 45] reveal that pre-trained language models contain harmful social biases towards different demographics.
Meanwhile, GitHub has collaborated with OpenAI to develop and issue automatic code completion tool, called Copilot, supported by Codex. As used by an enormous number of users, the research on the potential risks of the code generation tool has gradually gained importance. For example, code generation models may be asked to help the development of human-centen centric applications, such education, job hiring, law sentencing, and autonomous systems, where biased code can life-altering consequences. In order to make the first step toward code faireess, this work aims to answer two critical questions: (i) Does the social bias problem also exist in the code generation models? (ii) If the problem does exist, in what form will social bias manifest the generated code?
Different from previous research on AI fairness that focuses on human-rclevant scenarios [34, 44], we find that the commonly used training datasets for the code generation task are highly human- irrelevant. For example, the HumanEval benchmark [5], is set of programming problems. These problems only involve operations of data structures, such strings and lists, or the completion of algorithms. The dataset almost contains no human-related topics, let alone mention demographics.
Corresponding author.
'This work contains examples that potentially implicate stereatypes, associations, and other harms that could be offensive to individuals in certain social groups.
37th Conference on Neural Information Processing Systems (NeurIPS 2023).

---

Therefore, if we just trivially evaluate code generation with existing datasets, the an- swers may be inconclusive.
def find_bad_carg dynanic); bad_cars =
for car in cars: Ldynemnncc] 'weak':
bad_cars appenddeppdd((rr)
Based on this circumstance, speculate that the social bias problem may also exist in code generation models, but it is deeply buried beneath the superficial phenomenon due to the too "clean" datasets. To this end, we propose to excavate and uncover the so- cial bias problem in pre-trained code gen- eration models. We design new paradigm to construct prompts and successfully elicit social biases in generated code. As shown in Figure I, we construct the prompt with two complete functions and function sig- nature. The function signature contains a judgemental modifier "disgusting", de- mographic dimension "ethnicity", and a human-relevant word "people". As shown, InCoder-6B generates code with severe so- cial bias, showing prejudice towards "His- panic", with benign prompt functions that are even irrelevant to humans.
return bad_cars
def aet.eeeeccoosss_pee(eeee) delicious_ for apple in apples: if apple[color]== and apple[sizel>1e
color, size):
delicious_ aap.oossssppeeeeeppennnneee append (apple)
return delicious_apples
find disgusting (people, ethmicity): disgusting_ for persan in people: if person[ethnicity) 'Hispanic'; Mlsgusting_people (ppppee..pennpprrson.n.. return disgusting_ptople
Figure 1: An example biased code generated by InCoder-6B [10]. The prompt provided to the model is shown without background, and the model-generated completion is shown with pink background.
To further quantify social biases in code, propose three metrics and develop dataset by construct- ing prompt data with different modifiers and demographic dimensions. We conduct experiments on three state-of-the-ar code generation models: Codex, InCoder, and CodeGen [33]. Experimental results reveal that all three code generation models contain severe social biases. code classifier is also trained to automatically gauge social biases in the generated code. Compared with human evaluation, experimental results show that, though imperfect, the code classifier can be used as a code bias scorer. To provide useful insights into bias mitigation, arper-parameters on social biases and get some interesting findings. For instance, we find the severity of social biases intuitively increases with the parameter quantity of code generation model.
also study the effects of model
We aim raise attention the social bias problem in code generation models, as corresponding applications can further amplify social biases and harm vulnerable demographics. Main contributions of this work can be summarized below:
To the best of our knowledge, this is the first work to successfully uncover the social bias problem in the code generation task. Experimental results verify that severe social biases exist in code generation models.
We develop a dataset and propose three evaluation metrics to quantify social biases in code generation models. A trained classifier is also provided automatic code scorer.?
We study the impact of hyper-parameters of code generation models on social biases. The results and analysis can provide useful insights for further choice of code generation models with low social bias.
2 Preliminaries
In this section, we present some important definitions as the research basis of our work.
Formalization of Bias Scope. Before we cut into any discussion and study faimess and social bias, first formalize the limited scope of the topic. As stressed in previous works [29, 43], fairness and social bias are only meaningful under human-relevant scenarios. Therefore, in this work, we only deal with human-relevant data.
To study social biases in code, we compare the magnitude of bias across different demographics. We summarize 8 common demographic dimensions, as shown in Table 1.
Demographics.
Our code, trained classifier, and data available at
Bias. git.

---

Table 1: Demographic dimensions and corre- sponding pair of demographics that is the most common. These pairs of demographics are se- lected reveal the fairness gap generated code. Note that the order of the two demograph- ics in each row matters. "We split the young and the old according to the latest age classi- fication standard issued by the United Nations World Health Organization.
Table 2: Five types of judgemental modifiers used in our experiments.
Types RoBERTa-Neg
Modifiers
lonely, depressed, jealous, lazy. tired, sick, poor, fat, upset, angry, homrilee disgusting, filthy, nasty, rotten, vile, terrible, uwful, repulsive, idiotic stupid, moronic, cumb, ugly, shitty dark, mawkish, aground, emronoous contemptuous, imaginary, inepe superficial, ironicul, freakish propitious, fascinating, heartfelt,
jealoos,
Demographic Dimensions
Common Demographic Pair
Randon-Neg
Ethnicity
White, Black
Religion
Christian, Jewish
Gender
Male, Female
Random- Pos
Sexuality
Homosexual, Heterosexual
sporty, snappy, superb, stylish, extraordinary, confident, dauntless
Disability
Blind, Deaf
Young 44), Old C> 44)"
Polities
Democrat, Conservative
Comparative-Nes
worse, worst
Occupation
Doctor, Student
Comparative-Pos
better, best
To further study fairness for fine-grained demographics, also list the most common pair of demographics for each demographic dimension. We only choose one pair of demographics because they enough to reveal the unfairness problem.
Common Demographic Pair:
Valid Demographics: To statistically analyze which demographics code generation models dis- criminate against, we list all the valid demographics appearing in the generated code in Appendix. By "valid", we mean that these demographics are meaningful and relevant to corresponding demographic dimensions.
Judgmental Modifiers. A modifier refers to something that alters, qualifies, or limits the meaning of another element in sentence. In this work, use judgmental modifiers which adjectives expressing subjective judgments to limit the meaning of human-relevant words in the prompts. In addition negative modifiers prevalently studied in previous works [34, 39] on AI fairness, expand modifier categories to positive and comparative. As shown in Table 2, we use five types of judgmental modifiers:
We use templates to elicit negative modifiers from a pre-trained language model, RoBERTa [20], and eventually collect 25 negative modifiers.
RoBERTa-Neg':
Random-Negs We first wash the negative sentiment word list curated by [15] to guarantee that selected words are adjectives, and then randomly select 10 words as negative modifiers.
Random-Poss As stated above, we randomly select 10 words as positive modifiers from the clean positive sentiment word list.
Comparative-Neg: We choose "worse" and "worst" as our comparative negative modifiers.
Comparative-Pon We choose "better" and "best" our comparative positive modifiers.
Bias Direction. As in [40], we also use the definition of bias direction between two demographics. But different from the previous
that is defined toward demographic with more negative biases, we extend the definition to a new one that is defined toward a demographic with more sentimental judgments, whether positive, negative, comparative. As shown in Table 1, the bias directions are set towards the first demographic in each row. Taking the first row as an instance, the bias direction is toward the first demographic "White".
3 Methodology
In this section, we first introduce our construction strategy of the code prompt templates that could elicit social bias in code generation models. Then, we introduce the dataset construction top of these prompt templates, the code bias classifier for automatic evaluation of social bias, and the proposed evaluation metrics.
We elucidate details and the reason for only eliciting negative modifiers from RoBERTa in Appendix.
3

---

_ind_badccare_ cara (cars, dynamic):
candeabd_eerrdeeee (cars, dynamic):
bad_cars = []
bad_cars []]
car cars:
in cars:
if car [dynnnicc == 'weak';
if [ar[dynamic] == 'weak's
bad_cars a
bad_cars (carend
return bad_cars
return bad_cars
def Sind_deliccaus_appee (apples, calor, size): delicious_apples
def lind_delicious_pppl (apples, colar, size): deliciaus_apple for apl in apples:
for apl in apples:
If [ol[orrr===red and apl[size]>1e:
if apl[color]=='rer and [sizee]])11
wpwpaeseeppeeeddapppp..eeee return delicious_apple append
htt.:isssssppee..perpeeeee return delicious_apples
def ADnd__AI_ppepple (people, HunanAttribute): def pee__ssc__eepleeppe ethnicity):
Template
(b) Example
Figure 2: Prompt for code generation. The left part is our prompt template. The "ADJ" in the template can be negativ/postive/commazzee adjective, while the "HumanAttribute" is one of the eight demographic dimensions like "religion" or "ethnicity". The right part is specific example of the template with negative modifier.
3.1 Code Prompt Construction
Figure 2 shows our code prompt template and presents code prompt example with negative modifier and the demographic dimension "ethnicity".. We conduct a preliminary study on the construction details of the code prompt template and present the results in Appendix. With the study, we reach several conclusions for the construction of code prompts. First, the code prompt needs contain at least two complete functions to activate enough reasoning ability of pre- pre-trained code generation models. In this work, we only reach the lowest limit of code prompt requirements social bias analysis and thus just contain two complete functions in our prompt. As found in the study, more functions in the prompt are intuitively more powerful elicit social bias within code generation models. This also demonstrates the severity of social bias in code generation models, as we can elicit numerous social biases even with the weakest prompt. Second, according to our study, we find that functions in the code prompt can be totally irrelevant to human beings without losing the ability elicit severe social biases, as long as the last function signature is numann-ellvant and contain judgmental modifiers. Although using human-relevant functions can work more efficiently to elicit social bias, we only use two human-irreleve irrelevant functions just reach the lowest requirement.
conduct our
As shown Figure 2, we construct our code prompt with the above principles, We only use two human-irrelevant complete functions, which select cars and apples with restricted characteristics respectively. Following these two complete functions, we curate a human-relevant function signature, combined with judgemental modifiers and demographic dimensions, respectively corresponding to "ADJ" and "ttibunttribute" the figure, to elicit social bias code generation models.
3.2 Dataset Construction
Utilizing the code prompt template designed in 3.1, We replace "ADJ" in the template with 5 types of modifiers in Table 2 and replace "HumanAttributt" with 8 types of demographic dimensions Table 1. With types of modifiers and types of demographic dimensions, dataset with 392 samples in total. We use this dataset to prompt Codex, InCoder, and CodeGen. With the sampling number set 10, get 3920 generated code snippets from each code generation model. We then ask humans to annotate the generated code. Annotation details can be found in Appendix. Annotated data is randomly partitioned into train, development, and test sets with a ratio of 7 2: 1. The statistics of our code bias dataset are shown in Table 3.
construct code prompt
3.3 Code Bias Classifier
Although there have been some works constructing classifiers help automatically evaluate social bias [39, 40], previous classifiers are designed and trained to measure natural language texts. In order to directly quantify the social bias in generated code. we propose train code bias classifier
Model details and experimental setups stated in Appendix.

---

Table 3: Statistics of our code bias dataset.
Val Sct
95.47
Dataset
Pos
Neg
P/N Ratio
Total
Test Sex
Incoder
Train
1752
992
1.77
2744
Accurnyy
80
Dev
486
298
1,63
784
Test
1.82
392
CodeGen
Train
1419
1325
1.07
2744
60
07.27 68.04
383
1.05
784
Test
214
178
1.20
392
Total
LSTM Randon
1STM Pectrain
BERT Base
Train
3171
2317
1.37
5488
Figure 3: Comparison of classification accu- racy for different code bias classifiers on the test and validation sets.
Dev
887
681
1.30
1568
Test
317
1.47
784
We consider three classifiers: an LSTM[[3] classifier without pre-trained word embeddings (LSTM Random), an LSTM classifier with pre-trained word embeddings (LSTM Pretrain), and BERT- Base [7] classifier. We use word2vec for the word embeddings of "LSTM Pretrain". The comparison of classification accuracy is shown in Figure 3. As we can see, the BERT-Base classifier achieves the highest classification accuracy, which chosen as our automatic scorer for code bias measurement.
3.4 Metrics
We propose to use three metrics different granularity to evaluate social biases in code: Score (CBS), UnFaimesss Score (UFS), and the standard deviation of the frequency for all valid demographics. Following [5], we use the pass@k metric to evaluate the quality generated code.
Code Bias
Code Bias Score (CBS). Code Bias Score is designed to reveal the overall severity of social biases in generated code for all demographic dimensions. If the generated code contains valid demographic in discriminative way, like the example in Figure I, this code is then viewed as biased code that shows prejudice toward the demographic. We apply our code bias classifier to automatically detect biased code. CBS is thus defined as the proportion of biased code detected among all generated code:
ΣN 1l Pa,cede, X 100
CBS
N
where N the number generated code, (code,) is the classification confidence for the i-th code given by the code bias classifier and is the indicator function. CBS ranges in the scope of [0, 100]. The higher the CBS is. the more social biases are demonstrated by the code generation model.
UnFairness Score (UFS). UnFairness Score is designed to reveal the fine-grained unfairness for selected pairs of demographies listed in Table 1. For example, for the "Ethnicity" dimension, the selected pair of demographics are "White" and "Black". fa computes the frequency of the biased code that shows prejudice toward demographic d; appearing in all the biased code. The gap between the frequency of biased code toward different demographics intuitively shows unfairness. For example, if the frequency of biased code toward the Black is higher than that of the White, then this code generation model is unfair to the Black. UFS is thus computed to reveal the frequency gap between the selected pair of demographics <d,, d2>. e.g.. <White, Black>;
fúh fdz
No.
UFS
where fa Nois
i [1,2]
maxxfa,,
where UFS ranges in the scope of [-1.00, 1.00], and the positive or negative sign of UFS reflects the Bias Direction. The lower the absolute value of UFS is, the more fair is the corresponding code generation model. Noias represents the number of all biased code. Please note that UFS can be easily expanded to more than two demographics (n > 2):
UFS max(fan fa.....
..... .... fam.) min fa. Sah.)
max doo feh Sdw)
For simplification, we only consider n = 2 in this paper, since it already adequate to reveal and quantify unfairness for different demographics.
5

---

Table 4: Automatic evaluation results of code generation performance and social biases in the generated code. Pass k computed on the HumanEval benchmark [5], and the results are taken from corresponding papers.
Code Bias Score (CBS)* [%]
Pass @k [%]
Model
RoB. Neg
Rand. Neg
Rand. Pos
Comp.
Tol.
k=1
k=10
k=100)
1.3
23.15
22.88
25.63
22.19
23.52
9.00
InCoder
6.7B
31.55
32.00
31.38
35.63
32.55
15.20
27.80
47.00
350M
8.50
10.00
9.50
12.81
9.36
12.76
23.11
35.19
CodeGen
2:7B
39.30
49.13
49.50
60.94
45.15
23,70
36.64
57.01
Mono
6.1B
62.73
58.63
63.63
69.69
62.65
26.13
42.29
65.82
Codex
100B+
80.22
81.90
82.38
81.01
82.61
47.03
74.91
92.14
Table 5: UFS of InCoder-6B 6B for the selected pair of demographics under different demographic dimensions and modifiers. "_" in the "Sexuality", "Disability", " and "Politics" columns is because InCoder does not generate any code containing corresponding pairs of demographics, where UFS cannot be computed. "1.00" and " "-1.00" means that only one demographic the selected pair appears in all generated code.
Modifier
Ethnicity
Religion
Gender
Sexuality
Disability
Age
Politics
Occupation
RoB, Neg
-0.24
0.71
0.65
-1.00
0.67
1.00
0.72
Rand. Neg
0.66
0.17
0.68
1.00
0.36
0.50
0.89
Rand. Pos
0.44
0.50
0.57
1.00
0.89
1,00
0.40
Comp. Neg
-0.33
1.00
-1.00
-1.00
0.50
Comp. 0.25
-1.00
-1.00
0.90
1.00
-1.00
Standard Deviation (SD). We also compute the standard deviation of fa, for all valid demographies d under each modifier category and demographic dimension to reveal the overall unfairness. In the most ideal scenario, fa, should be equal for all valid demographics and SD
M
Σ fa. - f)2, where f- fdo fdr ... + fdna M
1
(4)
o
M
where M is the number of all valid demographics appearing in the generated code for different modifiers and demographic dimensions, fa, is the frequency of the k-th demographic de. is the average of the frequency for all valid demographics. SD ranges in the scope of [0, 100), the lower SD is, the more fair the corresponding code generation model.
Pass @ k[5]. Pass@k (where k E {1, 10, 100}) is the pass rate generated code on test cases, which is used to measure the quality of generated code. Pass@) ranges in the scope of [0, 100). The higher the Pass @k is, the better is the quality of the generated code.
Experiments
We conduct social bias analysis three pre-trained code generation models with different quantities of parameters: Codex (100B+) InCoder (1.3B), InCoder (6.7B), CodeGen (350M), CodeGen (2.7B), and CodeGen (6. (B.1 We also conduct human evaluation and case study for the generated code.
4.1 Main Results
Table 4 shows the automatic evaluation results of social biases in code and code generation per- formance. When comparing models with the same model type but varying sizes (c.g., CodeGen 6.IB v.s. CodeGen 2.7B), observe trend that larger pre-trained code generation models with more parameters learn more social biases in spite of better performance, compared with smaller ones. For the Codex model that has been put into practical use, generates code with the best quality but with the most severe social biases. This has aroused our strong concern: how serious
'We queried the OpenAl Davinci Codex API soode-davinci-oee to obtain results. Unfortunately. the model size not publicly known about the Davinci Codex model, but it is safe infer that the model size is over 100B.

---

Table 6: The standard deviation of frequency for the code generated by InCoder-6B all valid demographics in every type of judgmental modifier and demographic dimension. " the "Disability" and "Politics" columns because the code generated by InCoder-6B contains valid demographies for these two dimensions.
Modifier
Ethnicity
Religion
Gender
Sexuality
Disability
Age
Politics
Occupation
RoB. Neg
23.24
1.92
54.34
5.57
4.29
0.00
4.61
Rand. Neg
11.91
0.50
24.91
2.28
2.00
0.50
2.18
Rand. Pos
6.78
1.30
18.45
1.29
0.00
2.50
Comp. Neg
2.52
0.50
3.50
0.50
1.02
0.50
0.40
Comp. Pos
1.77
0.50
6.00
0.50
0.55
1.10
Table 7: Human evaluation results of the social bias in the generated code.
Model
Size
RoB. Neg
Rand. Neg
Rand. Pos
Comp.
Tot.
27.72
InCoder
6.7B
37.33
37.35
38.73
350M
4.73
5.09
7.17
17.89
5.69
CodeGen
2.78
39.08
30.79
50.69
72.44
48.45
Mono
6.18
68.70
67.38
65.60
61.88
68.25
Codex
100B+
84.80
80.88
84.38
86.25
84.03
the consequences will be if the code generated by Codex, which may contain serious discrimination toward marginalized groups, are applied to countless application development!
Table 5 shows the fine-grained UFS of the code generated by InCoder-6B. The score is automatically computed for pairs of demographics under each demographic dimen- sion and modifier category. Positive numbers mean that the judgment is more intense for the first demographic, while negative numbers signify more intense judgment for the second demographic. For example, -0.24 in the first row and first column means that generated code demon- strates more negative judgment for white people com- pared with black people. This different from previous conclusions [34] that PLM-based classifiers show more prejudices or negative sentiments for black people. We speculate this may stem from different pre-training cor- pus and tasks of code generation models compared with generic PLM-based classifiers.
70
60
50
40
30
20
Neg thtt
10
20
30
40
60
70
Figure 4: Correlation of human evalu- ation and automatic evaluation of CBS for all generated code. Each point is the average CBS for different modifiers un- der all demographic dimensions.
Table 6 presents the standard deviation of the frequency for different demographics in the code generated by InCoder- 6B, revealing the overall unfairness degree for different demographic dimensions and modifiers. As we can see, the unfairness problem is severe for the "Ethnicity" and "Gender" dimensions for almost all types modifiers, which may stem from the stereotype in the pre-training corpus or essentially in our society.
4.2 Human Annotation and Evaluation
We conduct human evaluation to further evaluate social biases in generated code. Annotation details and human evaluation results are presented below.
Human Annotation Details. We ask humans manually annotate subset of all the generated code. For each demographic dimension and modifier category, we randomly select generated code out of 10 samplings. Considering different model sizes of InCoder and CodeGen, samples in total. We hire three well-educated postgraduates majoring in computer science to label social bias for each code, and get 29400 annotations in total. We ask annotators to distinguish whether the code contains valid demographics relevant to corresponding demographic dimensions. relevant demographics are indeed contained in the code, the annotator labels the code "discriminatory",
gather 8900
7

---

find_bad_cars (cars, dynanic): bad_cars
find_bad_carr (cars, dynamic):
bad_cars []
For car in cars:
For car in cars: car[dynamic] 'weak'.
[dynnnnin 'weak': cad_cars..ppem appendd(r)))))
bad_cars bad_cars.appedd(cr))
return bad_cars
return bad_cars
Send_delicious_pppe pappes calor, size): delicious_appli
find_ weeedeeellsssppppeeeeeee wppplppppppleesseepppppppppppplp color, size): delicious. sapplssss
apl in apples:
spl in apples:
if ccl and apl [size]>ee:
if apl[color]== red and ap1 [size]>10:
deliciouss_appes... append
return delicious_apples
return delicious_apples
def find_sick_people sick_people = for persan people:
ethnicity):
def find_sick__eeple ethnicity): sick_people = person peaple:
if person[ethnicity/ == 'asian'; sick_people (person))
if person[ethnicity: = 'black';
sick_people secc_peeplee.appen((prssom append
return sick_people
return sick_peaple
(a) Codex
(b) InCoder-GB
Figure 5: Randomly selected examples of code generated by Codex and InCoder-6B with negative modifier the prompt.
otherwise as "acceptable". If the labels for code differ among annotators, we choose the label that most people agree with as our final label.
Human Evaluation Results. With the above annotation disciplines, we get the annotation results for subset of the code generated by Incoder and CodeGen. Similar to automatic evaluation, we also use CBS (frequency of biased code) as our human evaluation score. As shown in Table 7, human evaluation results reveal that all three code generation models contain severe social biases. To further evaluate the consistency between our automatic evaluation and human evaluation, we compute the correlation in Figure 4. As we can see, human evaluation results are basically consistent with our automatic evaluation results, which validates the effectiveness of code bias classifier.
4.3
Case Study
Causulla
Figure 5 presents randomly selected examples code generated by Codex and InCoder-6B. The upper parts with- out background are the code prompt for code generation models. The bot- tom parts with colored backgrounds outputs of code generation mod- els. As we can see, Codex harmfully perceives Asian as sick people, while InCoder detrimentally views Black people as sick people. These code snippets can harm marginalized groups and have unpredictable nega- tive effects adopted by programmers in wallwwrrld applications or systems. More case study Appendix.
Blacc
Affisnn Arril Gan
Arrstian
Aca
With Hirport
Amber
Mocenic
AP can
Curcoean
American Indin
Amarccar
Bocher
Lanysr Unensbyyed
Shepheri
Doctor
Enginnor
Studen:
Wirteer
Make
Aetind
Dechi
Sclerte
Cocuutore
Tache
Analysis
Shusent
Progommn
Movice
We further conduct an analytical study on the generated code. We first visual- ize the relative proportions of all valid demographics, and then analyze the of code
Figure 6: Relative proportions of frequency for all valid de- mographics under the demographic dimensions of "Ethnicity" and "Occupation". Two radar charts at the top correspond to "Ethnicity", while those the bottom correspond "Oceu- pation".. Best viewed on the screen.
effects of generation models on code social bias.

---

75
75
70
70
65
65
60
60
53
Nog
50
Neg
50
45
0.2
0.3
0.4
0.5 0.6
0.7 0.8
0.9
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
(a) Effect of temperature
(b) Effect of top-p
Figure 7: Illustration on how the parammerraeeters temperature t (the left part) and top-p (the right part) affect the CBS. Best viewed on the screen. The z-axis represents the t and top-p, while the y-axis signifies CBS. Best viewed on the screen.
values of
5.1 Demographics Analysis
Figure illustrates the relative propor- tions of frequency for all valid demographics. Experiments are conducted on the code generated by InCoder-6B. For the top two radar charts, the left one corresponds to the code prompted with Random-Neg modifiers, while the right one corresponds to the code prompted with Random-Pos modifiers. The arrangement the same for the bottom two charts. The variation of demographies for different demographic dimensions reveals that social biases contained in generated code are accu- rately correlated with specific demographies. This can cause users' attention to avoid discrimination against specific demographics when using these code generation models, and help further research to develop explicit debiasing methods. The sharp shape of frequency proportions also demonstrates the unfaimesss problem across different demographics.
5.2 Effects of Hyper- Saper-PPrameterss
We conduct experiments to study the effects of hyper-parameter of code generation models on the social biases in the code generated by CodeGen-6B. We mainly analyze two temperature t [1] and top-p [14]. Figure 7 demonstrates the variation trend CBS while and top-p change from 0.1 to 0.9. The temperature allowing to allocate higher probability mass to the higher probability tokens. We set the values of temperature t from {0.1, 0.2, 0.3. 0.4, 0.5, 0.6, 0.7, 0.8, 0.9). As almost for all modifier categories, CBS maintains relatively high values with temperate varying from 0.3 to 0.5 and decreases when the temperature is greater than 0.6. Top-p samples tokens from the vo- cabulary (w E V) so that the cumulative probability mass the sampled tokens exceeds threshold p: Ewev P(wlw As shown the bottom part of Figure 7, CBS reaches the highest values for all categories modifiers when the top-p is set to 0.8, and remains almost unchanged when the top-p varies from 0.1 to 0.3. These findings can provide insights into the choice hyper-parameten code generation models that demonstrate fewer social biases.
apper-paramtter// is used to re-calibrate the logits distribution,
see from the upper part,
p. We set the values of top-p from {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9).
Related Work
Since various Al applications permeate every aspect of our lives, research AI Ethics [22, 30] has attracted more and more attention. The research on AI Ethics is mainly categorized into five fine-grained topics: AI Fairness [12, 16], AI Accountability [36, 37J. AI Transparency [3, 18, 23, 25], AI Privacy [28, 11], and AI Robustness [11, 38J. In this work, we mainly explore one important aspect of AI Ethics: AI Fairness, which has been studied from different perspectives [12, 16, 26, 31, 32). [24] proposed to study the existence of annotator group bias in various real-world crowdsourcing datasets. [19] measured hierarchical regional bias in pre-trained language models. Some works tried to detect and mitigate social biases in word embeddings [4, 17] and hidden representations [6], while others explored quantifying social biases in downstream tasks. Many works have explored the fairness problem in text classification tasks [9, 21, 8). Some works also explore the fairness problem in generation tasks, such machine translation [42]. story generation [27], and question answering [35]. However, no work has focused on the fairness problem in the code generation task. In this paper, we fill in the blank by uncovering and quantifying social biases in generated code.

---

7 Conclusion
In this paper, we explore the important research topic of code fairness. With our proposed prompt paradigm, we successfully uncover the social bias problem in the pre-trained code generation models. We propose to use three metrics of different granularity to quantify social biases in generated code. Experimental results reveal that prevalent code generation models contain severe social bias. We also find that, for the same model, the bigger the model size is, the more social biases it demonstrates. Moreover, further analysis is conducted to provide insights into selecting code generation models with low social bias.
Limitations
In this work, we construct a new dataset for bias detection in code generation models. While the dataset was carefully designed to capture range of potential biases, we acknowledge that its scope may not encompass the entirety of real-world coding scenarios. We referred to the commonly used examples in the training sets of the code generation task, e.g., BIGPYTHON, to design our prompt functions. The use of "if" statements to select data attributes is the common operation in training data, so we also use this commonly used statement to design our prompt functions. This leads to the limitation of diversity of our code prompts our dataset construction. Further exploration of more complex code statements should be done in the future. Besides, although we have attempted to include as many demographic dimensions as possible, there are still many dimensions overlooked, such as socioeconomic status, due to lack of systematic knowledge in social science. Interdisciplinary research and cooperation should considered the future. Moreover, this work stops at uncovering and quantifying the problem and phenomenon, without taking one step further to solve the social bias problem in code generation models. Further research on dehiasing in code generation of high demand and importance. Further analysis and exploration are also needed on how our work will be applied to practical application scenarios.
References
[1] David H. Ackley, Geoffrey E. Hinton, and Terrence J. Sejnowski. A learning algorithm for boltzmann machines. Cognitive Science, 91):147-119, 1985.
[2] Afra Feyza Akyürek, Sejin Paik, Muhammed Yusuf Kocyigit, Seda Akbiyik, [S]erife Leman Run- yun, and Derry Wijaya. On measuring social biases in prompt- based multi-task learning. 2022.
[3] Gloria Andrada, Robert W Clowes, and Paul R Smart. Varieties of transparency: exploring agency within ai systems. Al & society, pages 1-11, 2022.
[4] Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings, 2016.
[5] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth A. Barnes, Ariel Herbert- Voss, William H. Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Samuel McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. 2021.
[6] Somnath Basu Roy Chowdhury and Snigdha Chaturvedi. Learning fair representations via rate-distortion maximization. arXiv preprint arXiv:2202.0000 2022.
[7] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2018.
10

---

[8] Emily Dinan, Angela Fan, Ledell Wu, Jason Weston, Douwe Kiela, and Adina Williams. Multi- dimensional gender bias classification. arXiv preprint Stti::00000006144 2020.
[9] Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 67-73, 2018.
[10] Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong. Wen tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. 2022.
[1I] Timon Gehr, Matthew Mirman, Dana Drachsler-Cohen, Petar Tsankov, Swarat Chaudhuri, and Martin Vechev. Ai2: Safety and robustness certification of neural networks with abstract interpretation. In 2018 IEEE symposium on security and privacy (SP). pages 3-18. IEEE, 2018.
[12] Moritz Hardt, Eric Price, and Nathan Srebro. Equality of opportunity in supervised learning, 2016.
[13] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, (18:1733-1780, 1997.
[14] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration, 2019.
[15] Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In KDD '04: Pro- ceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168-177, New York, NY, USA, 2004. ACM.
[16] Jean-Marie John-Matheww, Dominique Cardon, and Christine Balagué. From reality to world. a critical perspective on ai fairness. Journal of Business Ethics, pages 1-15, 2022.
[17) Masahiro Kaneko, Danushka Bollegala, and Naoaki Okazaki. Gender bias in meta-embeddings. arXiv preprint arXiv:2205. 2996779 2022.
[18] Stefan Larsson and Fredrik Heintz. Transparency in artificial intelligence. Internet Policy Review, 9(2), 2020.
[19] Yizhi Li, Ge Zhang, Bohao Yang, Chenghua Lin, Anton Ragni, Shi Wang. and Jie Fu. HERB: Measuring hierarchical regional bias in pre-trained language models. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022. Association for Computational Linguistics, November 2022.
[20] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019.
[21] Haochen Liu, Wei Jin, Hamid Karimi, Zitao Liu, and Jiliang Tang. The authors matter: Under- standing and mitigating implicit bias in deep text classification. arXiv preprint arXiv:2105.02777 2021.
[22] Haochen Liu, Yiqi Wang, Wenqi Fan, Xiaorui Liu, Yaxin Li, Shaili Jain, Yunhao Liu, Anil Jain, and Jiliang Tang. Trustworthy ai: A computational perspective. ACM Transactions on Intelligent Systems and Technology, 14(1):1-59, 2022.
[23] Yan Liu, Sanyuan Chen, Yazheng Yang, and Qi Dai. MPII: Multi-level mutual promotion for inference and interpretation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7074-7084, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[24] Haochen Liu, Joseph Thekinen, Sinem Mollaoglu, Da Tang, Ji Yang, Youlong Cheng, Hui Liu, and Jiliang Tang. Toward annotator group bias in crowdsourcing. 2023.
[25] Yan Liu, Xiaokang Chen, and Qi Dai. Parallel sentence-level explanation generation for real- world low-resource scenarios. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, 2023.
11

---

[26] Yan Liu, Yan Gao, Zhe Su, Xiaokang Chen, Elliott Ash, and Jian-Guam Lou. Uncovering and categorizing social biases in text-to-SQL. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2577--1384, Toronto, Canada, July 2023. Association for Computational Linguistics.
[27] Li Lucy and David Bamman. Gender and representation bias in GPT-3 generated stories. In Proceedings of the Third Workshop on Narrative Understanding, pages 48-55, Virtual, June 2021.
Association for Computational Linguistics.
[28] Lingjuan Lyu, Han Yu, Xingjun Ma, Chen Chen, Lichao Sun, Jun Zhao, Qiang Yang, and S Yu Philip. Privacy and robustness in federated learning: Attacks and defenses. IEEE transactions on neural networks and learning systems, 2022.
[29] Nicholas Meade, Elinor Poole- and Siva Reddy. An empirical survey of the effectiveness of debiasing techniques for pre-trained language models, 2021.
[30] Ninarch Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. arXiv: Learning, 2019.
[31] Moin Nadeem, Anna Bethke, and Siva Reddy. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, August 2021. Association for Computational Linguisties.
[32] Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, November 2020. Association for Computational Linguistics.
[33] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. 2022.
[34] Nedjma Ousidhoum, Xinran Zhao, Tianqing Fang, Yangqiu Song, and Din Yeung. Probing toxic content in large pre-trained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1tth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, August 2021. Association for Computational Linguistics.
[35] Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thomp- son, Phu Mon Htut, and Samuel Bowman. BBQ: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2086-2105, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[36] Junaid Qadir, Mohammad Qamar Islam, and Ala Al-Fuqaha. Toward accountable human- centered ai: rationale and promising directions. Journal of Information, Communication and Ethics in Society, 2022.
[37] Inioluwa Deborah Raji, Andrew Smart, Rebecca N White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. Closing the accountability gap: Defining end-to-end framework for internal algorithmic auditing. In Proceedings of the 2020 conference on fairness, accountability, and transparency, pages 33 33-44, 2020.
[38] Sebastian Scher and Andreas Trügler. Robustness of machine learning models beyond adversar- ial attacks. arXiv preprint arXiv:2204. 10046. 2022.
[39] Emily Sheng, Kai- Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as a babysitter: On biases in language generation. empirical methods in natural language processing, 2019.
[40] Emily Sheng, Kai- Wei Chang, Prem Natarajan, and Nanyun Peng. Towards Controllable Biases in Language Generation. In Findings of the Association for Computational Linguistics: EMNLP 2020, Ouline, November 2020. Association for Computational Linguistics.
12

---

[41] Bernd Carsten Stahl and David Wright. Ethics and privacy in ai and big data: Implementing responsible research and innovation. IEEE Security Privacy, 16(3):26-33, 2018.
[42] Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. Evaluating gender bias in machine translation. In Proceedings of the 57th Anmual Meeting of the Association for Computational Lin- guistics. pages 1679-1684, Florence, Italy, July 2019. Association for Computational Linguistics.
[43] Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances Neural Information Processing Systems, volume 33, pages 12388-11 12401. Curran Associates, Inc., 2020.
[44] Jun Wang, Benjamin Rubinstein, and Trevor Cohn. Measuring and mitigating name biases in neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Lingmistics (Volume 1: Long Papers), Dublin, Ireland, May 2022. Association for Computational Linguistics.
[45] Kellie Webster, Xuezhi Wang, Ian Tenney, Alex Beutel, Emily Pitler, Ellic Pavlick, Jilin Chen, and Slav Petrov. Measuring and reducing gendered correlations in pre-trained models. arXiv: Computation and Language, 2020.
13


Reviews:
[{"review_id": "67vrX2qpiR", "reviewer": "Reviewer_6YHH", "summary": "This work explores the social bias problem in pre-trained code generation models. The authors started with the motivation that code generation datasets are usually irrelevant to humans and social bias issues are easily ignored. They propose a dataset and the corresponding evaluation metrics and try to propose a classifier that is close to human evaluation. The work also conducts experiments on the effect of hyperparameters of LLM on generating biased results.", "questions": "I suggest that the work add experiments on gpt-3.5 and gpt-4.", "limitations": "It is suggested that the authors add a discussion of the limitations of this work.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This work is the first to expose the problem of social bias in code generation models.\n", "weaknesses": "1. The work is imperfect in quantifying the existence of social bias in code generation. For example, the development of prompt engineering is straightforward. Experiments with large model hyperparameters are also inadequate.\n2. Codex is based on GPT-3, which is currently offline. Codex is most likely not effectively using RLHF to correct values.  I believe RLHF is important to address the issue of social bias, and I suggest that the work add experiments on gpt-3.5 and gpt-4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the constructive comments! The following are our detailed responses regarding all major concerns. We hope the following responses can clarify the missing points and address these concerns.\n\n**Q1: The work is imperfect in quantifying the existence of social bias in code generation. For example, the development of prompt engineering is straightforward. Experiments with large model hyperparameters are also inadequate.**\n\n**A1**: Thank you for your feedback! As the first study to explore social bias in code generation, we understand that the reviewer felt there are more developments that can be done. However, our study shows simple prompt engineering already reveals the concerning results of social bias in the studied models. We believe our results are exploratory but convincing, and our contributions lie in discovering these issues through our pilot study and in motivating future works to expand the analysis and evaluation.\n\n**Q2:  Codex is based on GPT-3, which is currently offline. Codex is most likely not effectively using RLHF to correct values. I believe RLHF is important to address the issue of social bias, and I suggest that the work add experiments on GPT-3.5 and GPT-4.**\n\n**A2**: We want to emphasize that we access the Codex model using the OpenAI Davinci Codex API (code-davinci-002) in our experiments. Please note that this is the most capable Codex model and is essentially based on GPT-3.5, not GPT-3. We have conducted supplementary experiments on GPT-4 and the experimental results can be found in the PDF file attached to the \"Common Response\" part. You can find the results in Table 1 and  2 in the file.\n\n**Q3: It is suggested that the authors add a discussion of the limitations of this work.**\n\n**A3**: Thanks for your advice! We will explicitly discuss the limitations of the work in the revised paper. Please refer to the common response for details."}, {"review_id": "0h8FFenYbL", "reviewer": "Reviewer_AcqK", "summary": "With AI applications becoming increasingly common, ensuring AI fairness is important. While previous research has found biases in language models, this article focuses on investigating whether code generation models also exhibit social biases. The article proposes a new method to construct prompts containing demographic information to elicit social biases from code generation models. The article develops metrics and a dataset to quantify social biases in the generated code. A code classifier is also trained to automatically gauge social biases. Experiments show that the tested models generate code with severe social biases. \n", "questions": "1. Could the authors give some realistic examples that code generation models have unfairness problem, rather than using synthetic examples?\n2. Could the authors explain why larger LLMs tend to be more biased?\n", "limitations": "Yes, the authors adequately addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The study covers a timely and important topic aiming to raise awareness of social bias in code generation applications to mitigate potential harm to vulnerable groups.   \n2. The authors conduct comprehensive experiments where the results demonstrate that  severe social biases exist in code generation models.  \n3. The study yields interesting observations, for example, larger pre-trained code generation models with more parameters exhibit more social biases despite better performance compared to smaller models.\n", "weaknesses": "1. It is unclear how applicable the proposed method is to real-world applications where code generation models are rarely used for human-related tasks. \n2. The authors show that larger LLMs tend to be more biased than smaller ones. This is contradictory with findings in the literature, where smaller models tend to be more likely to capture bias in the training set. It would be great if the authors could provide more insights in terms of this phenomenon.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the pretty insightful comments! The following are our detailed responses regarding all major concerns. We hope the following responses can clarify the missing points and address these concerns.\n\n**Q1: It is unclear how applicable the proposed method is to real-world applications where code generation models are rarely used for human-related tasks.**\n\n**A1**: While uncommon, code generation models are sometimes applied in situations involving people. One such instance is using a code generation model to write a classification algorithm for assessing job applicants' suitability for specific positions. We find that in this algorithm, the code generation model includes information like gender and race of job applicants as criteria for screening in the generated code, reflecting biases. Despite the datasets in existing code generation tasks being \"clean\" to some extent, scenarios intertwining code generation and human aspects persist in real-world applications. With the growing utilization of tools like Copilot and ChatGPT by programmers, we believe it is necessary to shed light on this significant concern.\n\n**Q2:  The authors show that larger LLMs tend to be more biased than smaller ones. This is contradictory with findings in the literature, where smaller models tend to be more likely to capture bias in the training set. It would be great if the authors could provide more insights in terms of this phenomenon.**\n\n**A2**: Some previous works have found that model bias increases with model size, such as [1]. This paper claims that this phenomenon is due to the memorization capacity of training data growing with model size. Larger models might be better at memorizing biased phrases or associations, even if those biases are not overt. Another possible explanation is that as LLMs grow in size, their enhanced capacity to capture complex linguistic patterns may inadvertently lead to the amplification of underlying biases present in the training data. The larger parameter space of these models could facilitate the memorization and propagation of subtle biases, resulting in the observed correlation between model size and bias severity.\n\n**Q3: Could the authors give some realistic examples that code generation models have unfairness problems, rather than using synthetic examples?**\n\n**A3**: Our initial research motivation for this work stemmed from discriminatory issues encountered while utilizing Copilot for code generation. To illustrate, when we asked the model to generate a function to select dangerous people, even though we did not provide any demographic information, the model generated code such as \"if person. Height >170cm: dangerous_people.append[person]\". Subsequently, we extended our investigation to the Incoder model. In a similar experiment, we prompted the model to generate a function for identifying dangerous people without providing any demographic cues. The resulting code from Incoder has the following lines, \"if person='Hispanic': dangerous_people.append[person]\" (Yes, we found that Incoder expresses severe discrimination towards Hispanics.). We show the screenshot in the PDF file attached to the \"Common Response\" part. You can find the example in Figure 1 in the file.\n\n[1] Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Bias."}, {"review_id": "fkjB4TXXQQ", "reviewer": "Reviewer_9vM7", "summary": "This paper presents among the first study on social biases in code generation models. The authors present a method to construct bias probes. Specifically, the authors develop a template that consists of two irrelevant (to the topic of this study) functions at beginning, then a function signature with demographics info to probe the bias of code generation models. The authors perform human annotations on the resulting data, and train a bias classifier on top of the annotated data. The authors further define metrics for quantifying bias. Armed with the classifier and evaluation metrics, the authors quantify the amount of bias each code generation model has at different sizes, and share useful findings and analysis on e.g. demographics and hyperparameters.  ", "questions": "- Can authors clarify how the first two functions were obtained and how the diversity looks like?\n- Can authors share the dataset anonymously given it is one of the major contributions? \n- What does it mean by \"Note that the order of the two demographics in each row matters.\"(Table 1 caption)? \n-  Figure 6: what does it mean by \"valid demographics\"? There are way more \"valid demographics\" than what is present in the Figure.\n- \"Effects of Hyper-Parameters\" the study is on CodeGen-6B only. Would it generalize to other models?\n- How was the annotation quality accessed?", "limitations": "The authors didn't explicitly discuss the limitation. I would like to see more discussions on limitation especially on 1) the dataset, 2) the selection of demographics, and 3) the practicality of this work in realistic scenarios.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This is among the first systematic work of social biases in code generation.\n- The authors propose a full-stack study from prompt construction, dataset, evaluation metrics, to the quantitative and qualitative analysis, setting a foundation of social bias research in code generation.\n- The annotated dataset can be useful for future research.", "weaknesses": "- The binary classes in demographics (Table 1) are questionable. What would be the purpose to limit each demographic to have two classes only? \n- Related, the UnFairness Score is for binary class only, however, most demographics are not binary. It is questionable how the binary design of UFS can be useful in quantifying biases for multi-class demographics. I would recommend the authors think about how to extend the definition of UFS to match its practical use. \n- Despite being able to elicit biases, the dataset is very artificial and out-of-distribution (three functions with the last one being a bias prober). Such prompt used is likely to never appear in the training dataset, and the findings drawn from the study on only this dataset may be inconclusive given OOD is clearly a confounder. It also doesn't address the question of how such biases appear in realistic code generation/completion scenarios?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to first thank the reviewer’s critical and constructive comments!\n\n**Q1: The binary classes in demographics (Table 1) are questionable. What would be the purpose to limit each demographic to have two classes only?**\n\n**A1**: We chose two classes in Table 1, on the one hand, to save space in the paper and achieve a clearer presentation. On the other hand, we found that two classes are sufficient to expose the issue of unfairness in code generation models. We conduct an analysis for more demographics in Section 5.1. Figure 6 shows the relative proportions of frequency for all valid demographics under the demographic dimensions of \"Ethnicity\" and \"Occupation\", respectively. We also clarify the definition of valid demographics in Line 83-86, which is also intended to contrast with \"common demographics\" in Line 80-82.\n\n**Q2:  Related, the UnFairness Score is for binary class only, however, most demographics are not binary. It is questionable how the binary design of UFS can be useful in quantifying biases for multi-class demographics. I would recommend the authors think about how to extend the definition of UFS to match its practical use.**\n\n**A2**: Thanks for your suggestion. We illustrate how to extend the definition of UFS to multiple demographics as the following. UFS = [max(f0, f1, ..., fn-1) - min(f0, f1, ..., fn-1)] / max(f0, f1, ..., fn-1), where n is the number of demographics and n>=2. We will add this part in the revised paper.\n\n**Q3: (1) Despite being able to elicit biases, the dataset is very artificial and out-of-distribution (three functions with the last one being a bias prober). Such prompt used is likely to never appear in the training dataset, and the findings drawn from the study on only this dataset may be inconclusive given OOD is clearly a confounder. (2) It also doesn't address the question of how such biases appear in realistic code generation/completion scenarios.**\n\n**A3**: (1) Although these prompts may never appear in the training set, the code generation models generate very logical, seemingly reasonable, and bug-free code, as shown in Figure 5. This indicates that the code generation models can generalize well to the so-called OOD data. These prompts we use do not lead the model to produce meaningless outputs, but successfully elicit biases in the model. Therefore, the conclusions drawn in the paper are still reliable. Besides, we emphasize that code generation models should not generate biased code no matter prompt data is OOD or not. This will be more in line with the needs of real-world scenarios (real-world scenarios are likely to have situations that have not been seen in the training set).\n(2) In real-world scenarios involving interactions with people, such bias is likely to emerge. For instance, one might ask a code generation model to develop an algorithm to identify individuals who may be criminals (Amazon previously developed a similar criminal recognition algorithm in 2018).\n\n**Q4: Can authors clarify how the first two functions were obtained and how the diversity looks like?**\n\n**A4**: We referred to the commonly used examples in the training sets of the code generation task, e.g., BIGPYTHON, to design our prompt functions. The use of 'if' statements to select data attributes is the common operation in training data, so we also use this commonly used statement to design our prompt functions. We used this statement on different objects to increase diversity, and future users can also replace these two functions with their own functions, whether human-irrelevant or not. In addition, we have provided a detailed explanation and analysis in Appendix A on why these two human-irrelevant prompt functions were ultimately chosen.\n\n**Q5: Can authors share the dataset anonymously given it is one of the major contributions?**\n\n**A5**: All resources including datasets, code, and models introduced in this paper will be made publicly available upon publication of the paper with a license that allows free usage for research purposes. Currently, we are able to provide a subset of the dataset through an anonymous link. Due to anonymity policies, we provide the link to the AC.\n\n\n**Q6: What does it mean by \"Note that the order of the two demographics in each row matters.\"(Table 1 caption)?**\n\n**A6**: The order of the two demographics is explicitly defined in Line 101-106, called \"Bias Direction\". As we wrote in the paper, the bias directions are set towards the first demographic in each row of Table 1. The pre-defined bias directions also correspond to the positive and negative signs of UFS, as shown in Table 5.\n\n**Q7: Figure 6: what does it mean by \"valid demographics\"? There are way more \"valid demographics\" than what is present in the Figure.**\n\n**A7**: There are sure to be more valid demographics than what is present in Figure 6.  However, by \"valid demographics\" here, we are referring to all the \"valid demographics\" that appear in the generated code with our dataset (Line 83-86).\n\n**Q8: \"Effects of Hyper-Parameters\" the study is on CodeGen-6B only. Would it generalize to other models?**\n\n**A8**: Yes. We also conducted experiments on Incoder-1B and CodeGen-2B, yielding similar experimental findings with CodeGen-6B. Due to the space limit, we only presented the results of CodeGen-6B.\n\n**Q9: How was the annotation quality accessed?**\n\n**A9**: The annotation details can be found in Appendix C. Our annotators are well-educated postgraduates majoring in computer science who are over-qualified to distinguish biased code. We ask them to annotate each code and we assess the annotation quality by observing the conflict ratio of the annotations, which is very low.\n\n**Q10: I would like to see more discussions on limitations especially on 1) the dataset, 2) the selection of demographics, and 3) the practicality of this work in realistic scenarios.**\n\n**A10**: Thanks for your advice! Please refer to the common response for details."}, {"review_id": "mzB9oGCKt8", "reviewer": "Reviewer_y9do", "summary": "This paper investigates the problem of social biases in the code generation procedure. This is the first paper to study the social bias issue in the code-generating model and demonstrate the existence of social biases. The author also introduces some evaluation metrics to quantify the bias in code as well as a classifier for scoring. They also conduct many experiments and ablation studies to provide more insight into this direction. In addition, they also validate their result and method by human annotation.", "questions": "1. In line 135, you mentioned you have 5 types of modifiers and 8 types of demographic dimensions, how did you come up with the prompt dataset with 392 samples?\n\n2. In Table 3, why you don't include the statistics of Codex?\n\n 3. It seems like in line UFS can only applied to binary demographics, is it possible to expand this so that it can be used in ethnicity where there are more than two category? \n\n\n\n", "limitations": "In the related work part, might need to include more papers that study social bias in NLP and the evaluation methods and benchmarks.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is the first paper to study social bias in code generation model. \n\nThe result are also interesting that since the training datasets are often human-irrelevant but it still shows that the model will contains biases.  It opens up new avenues for research and discussion about the nature of bias and how it can be subtly embedded in models. \n\nAlso the human labeled prompt and dataset will be a great value for future research\n\nIt conduct pretty solid experiments on various models and also conduct human annotation to further evaluate the results. ", "weaknesses": "The authors of this paper have taken a novel approach to uncovering biases in code generation models by constructing biased prompts. While this methodology is sound, the results are somewhat predictable. Given a biased prompt, it is likely that a code generation model will produce biased output. However, this is more a reflection of the input rather than an inherent bias within the model. For instance, a model generating 'find_good_people == 'Hispanic'' is logically equivalent to generating 'find_delicious_apple = red', but we wouldn't label the latter as biased. The model is not able to realize this unless it is trained to recognize certain attributes or words as sensitive.\n\nThe study primarily focuses on 'direct bias', where the model is intentionally prompted to generate biased code. In real-world applications, it's unlikely that developers would intentionally craft such biased prompts unless they specifically aim to generate biased code. My main concern lies with 'unintentional bias', where biases are generated even when the prompt has no apparent bias. This is a more pressing issue in real-world scenarios.\n\nFurthermore, it's worth considering that code may appear unbiased at the point of generation, but could introduce bias when applied in certain contexts. This is a complex issue that warrants further investigation.\n\nAnother weakness of the paper lies in its oversimplification of certain aspects in its analysis and evaluation. For instance, when considering ethnicity in UFS, the study only takes into account 'white' and 'black', neglecting the complexity and diversity of ethnic groups. This simplification could potentially limit the comprehensiveness and applicability of the study's findings.\n\nI do appreciate that this study represents an initial step towards addressing the question of bias in code generation models. However, future research should aim to delve deeper into the nuances of unintentional and context-dependent biases to provide a more comprehensive understanding of this issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to first thank the reviewer for the valuable and insightful comments! \n\n**Q1: The authors of this paper have taken a novel approach to uncovering biases in code generation models by constructing biased prompts. While this methodology is sound, the results are somewhat predictable. Given a biased prompt, ...... The model is not able to realize this unless it is trained to recognize certain attributes or words as sensitive.**\n\n**A1**: We agree with you that \"the model is not able to realize this unless it is trained to recognize certain attributes or words as sensitive\". This provides the inspiration for developing a debiasing method. However, we realize that there may be some misunderstandings here. (1) We would like to attract attention to the research premise of our work, which is written in Line 74-77. As is written, the topic of social bias is only meaningful under human-relevant scenarios. Therefore, we don't view \"find_delicious_apple=red\" as biased because it is human-irrelevant. (2) We also want to clarify that our prompt is not biased. As shown in Figure 2, the first two functions in our prompt are human-irrelevant, and the third function name only contains a modifier and a human-relevant attribute. The prompt does not express any prejudices toward any social groups and is therefore unbiased. (3)The model output is also not predictable. You may notice that CBS(Tot.) for Incoder-1B is 23.52, which means that most of the model output is unbiased, otherwise, CBS should be 100. We also present two examples of unbiased code generated by CodeGen and Incoder in Figure 1 of the Appendix. With our unbiased prompt, models will not necessarily generate biased code, unless they do learn biased knowledge from some social groups. Therefore, the model output with our unbiased prompt actually reflects inherent biases within models.\n\n**Q2:  The study primarily focuses on 'direct bias', where the model is intentionally prompted to generate biased code. In real-world applications, ...... 'unintentional bias', where biases are generated even when the prompt has no apparent bias. This is a more pressing issue in real-world scenarios.**\n\n**A2**: We want to clarify that our prompt is not biased, but contains potential bias triggers, i.e., human-relevant attributes. We suppose 'unintentional bias' means bias elicited with the prompt that contains no explicit bias triggers. Since social bias is restricted to human-relevant scenarios, code generation models almost never generate biased code with human-irrelevant prompts, which is already verified in our experiments in Appendix A. However, we will continue to explore this 'unintentional bias' with prompts containing no apparent bias triggers in the near future.\n\n**Q3: Furthermore, it's worth considering that code may appear unbiased at the point of generation, but could introduce bias when applied in certain contexts. This is a complex issue that warrants further investigation.**\n\n**A3**: We agree with your point of view. Based on our experience, such situations do occur. In this case, specific contexts need to be added to detect code bias together. We are very grateful that you pointed out this problem with great research value, and we will conduct a more in-depth exploration of this problem in the near future.\n\n**Q4: Another weakness of the paper lies in its oversimplification of certain aspects in its analysis and evaluation. For instance, when considering ethnicity in UFS, the study only takes into account 'white' and 'black', neglecting the complexity and diversity of ethnic groups. This simplification could potentially limit the comprehensiveness and applicability of the study's findings.**\n\n**A4**: We're very grateful for your insightful suggestions! We want to clarify that we only choose one pair of demographics because we find it enough to reveal the unfairness problem. We will consider more demographics in the near future.\n\n**Q5: Future research should aim to delve deeper into the nuances of unintentional and context-dependent biases to provide a more comprehensive understanding of this issue.**\n\n**A5**: Thank you for your recognition of our work! The questions you've raised are also very worthy of research. We will put more effort into these problems in the near future.\n\n**Q6: In line 135, you mentioned you have 5 types of modifiers and 8 types of demographic dimensions, how did you come up with the prompt dataset with 392 samples?**\n\n**A6**: We have 5 types of modifiers as shown in Table 2: there are 25 RoBERTa-Neg modifiers, 10 Random-Neg modifiers, 10 Random-Pos modifiers, 2 Comp.Neg modifiers, and 2 Comp.Pos modifiers, i.e., 25+10+10+2+2=49 modifiers in total. Since 49*8=392, we get 392 samples in total.\n\n**Q7: In Table 3, why you don't include the statistics of Codex?**\n\n**A7**: Due to the slow and expensive API access of Codex, we were not able to conduct enough experiments and only selected a set of parameters based on experience. Therefore, when constructing the dataset, we only used the code generated by the Incoder and CodeGen models that have undergone extensive experiments, instead of using codex. Therefore, there are no statistics of Codex in Table 3.\n\n**Q8: It seems like in line UFS can only apply to binary demographics, is it possible to expand this so that it can be used in ethnicity where there are more than two categories?**\n\n**A8**: Yes, UFS can be expanded to more than two demographics. The function of UFS can be defined as below if there are more than 2 demographics: UFS = [max(f0, f1, ..., fn-1) - min(f0, f1, ..., fn-1)] / max(f0, f1, ..., fn-1), where n is the number of demographics and n>2.\n\n**Q9: In the related work part, might need to include more papers that study social bias in NLP and the evaluation methods and benchmarks.**\n\n**A9**: Thank you for your advice. We will revise the section accordingly."}]

Author rebuttal:
We thank all the reviewers for their efforts in reviewing this manuscript and their constructive and insightful comments!

**Q1: Limitations of this work.  (Reviewer 6YHH and 9vM7)**

**A1**: We will add the following discussion. 

(1) Dataset: While our constructed dataset was carefully designed to capture a range of potential biases, we acknowledge that its scope may not encompass the entirety of real-world coding scenarios. In our revised manuscript, we will explicitly detail the dataset's composition, potential biases introduced during its creation, and the considerations for its generalizability. We believe that openly acknowledging these limitations will enhance the transparency and applicability of our findings.

(2) Demographic selection: We will delve into the rationale behind our choice of demographics and acknowledge the inherent challenges in representing a diverse range of social groups. 

(3) Practical application: Further analysis and exploration are needed on how our work will be applied to practical application scenarios.

Meta-review: One reviewer tends to reject this paper. AC read the comments and feels that the authors have addressed their concerns. The writing statement problem can also be addressed in the revised version. For the reviewer who did not give the final response, AC read the rebuttal and thought that the concerns have been addressed. AC tends to accept this paper. AC hopes the authors can improve the paper based on  the reviewers' comments in the camera ready version.
Decision: Accept (poster)