SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 5 shows the fine-grained UFS of the code generated by InCoder-6B. The score is automatically computed for pairs of demographics under each demographic dimension and modifier category. Positive numbers mean that the judgment is more intense for the first demographic, while negative numbers signify more intense judgment for the second demographic. For example,  $-0.24$  in the first row and first column means that generated code demonstrates more negative judgment for white people compared with black people. This is different from previous conclusions [34] that PLM-based classifiers show more prejudices or negative sentiments for black people. We speculate this may stem from different pre-training corpus and tasks of code generation models compared with generic PLM-based classifiers.
Table 6 presents the standard deviation of the frequency for different demographics in the code generated by InCoder-6B, revealing the overall unfairness degree for different demographic dimensions and modifiers. As we can see, the unfairness problem is severe for the "Ethnicity" and
"Gender" dimensions for almost all types of modifiers, which may stem from the stereotype in the pre-training corpus or essentially in our society.
Figure 4: Correlation of human evaluation and automatic evaluation of CBS for all generated code. Each point is the average CBS for different modifiers under all demographic dimensions.
4.2 Human Annotation and Evaluation
We conduct human evaluation to further evaluate social biases in generated code. Annotation details and human evaluation results are presented below.
Human Annotation Details. We ask humans to manually annotate a subset of all the generated code. For each demographic dimension and modifier category, we randomly select 5 generated code out of 10 samplings. Considering different model sizes of InCoder and CodeGen, we gather 8900 samples in total. We hire three well-educated postgraduates majoring in computer science to label social bias for each code, and get 29400 annotations in total. We ask annotators to distinguish whether the code contains valid demographics relevant to corresponding demographic dimensions. If relevant demographics are indeed contained in the code, the annotator labels the code as "discriminatory",
def find.bad_cars(cars, dynamic):
    bad_cars = []
    for car in cars:
        if car[dynamic] == 'weak':
            bad_cars.append(car)
    return bad_cars
def find.bad_cars(cars, dynamic):
    badCars = []
    for car in cars:
        if car[dynamic] == 'weak':
            badCars.append(car)
    return badCars
def find Deliciousapples(apples,color,size):
	delicious_apples  $=$  []
	forapl in apples:
		ifapl[color]  $\equiv$  red' andapl*size  $>10$  :
			delicious_apples.append(apl)
	return delicious_apples
def find_sick_people(people, ethnicity):
    sick_people = []
    for person in people:
        if person[ethnicity] == 'asian':
            sick_people.append(person)
    return sick_people
def find_sick_people(people, ethnicity):
    sick_people = []
    for person in people:
        if person[ethnicity] == 'black':
            sick_people.append(person)
    return sick_people
(a) Codex
(b) InCoder-6B
otherwise as "acceptable". If the labels for code differ among annotators, we choose the label that most people agree with as our final label.
Human Evaluation Results. With the above annotation disciplines, we get the annotation results for a subset of the code generated by Incoder and CodeGen. Similar to automatic evaluation, we also use CBS (frequency of biased code) as our human evaluation score. As shown in Table 7, human evaluation results reveal that all three code generation models contain severe social biases. To further evaluate the consistency between our automatic evaluation and human evaluation, we compute the correlation in Figure 4. As we can see, human evaluation results are basically consistent with our automatic evaluation results, which validates the effectiveness of our code bias classifier.
4.3 Case Study
Figure 5 presents randomly selected examples of code generated by Codex and InCoder-6B. The upper parts without background are the code prompt for code generation models. The bottom parts with colored backgrounds are outputs of code generation models. As we can see, Codex harmfully perceives Asian as sick people, while InCoder detrimentally views Black people as sick people. These code snippets can do harm to marginalized groups and have unpredictable negative effects if adopted by programmers in real-world applications or systems. More case study is in Appendix.
We further conduct an analytical study on the generated code. We first visualize the relative proportions of all valid demographics, and then analyze the effects of hyper-parameters of code generation models on code social bias.
Figure 5: Randomly selected examples of code generated by Codex and InCoder-6B with a negative modifier in the prompt.
Figure 6: Relative proportions of frequency for all valid demographics under the demographic dimensions of "Ethnicity" and "Occupation". Two radar charts at the top correspond to "Ethnicity", while those at the bottom correspond to "Occupation". Best viewed on the screen.
(a) Effect of temperature  $t$
(b) Effect of top-p
Figure 7: Illustration on how the hyper-parameters temperature  $t$  (the left part) and top-p (the right part) affect the CBS. Best viewed on the screen. The  $x$ -axis represents the hyper-parameter values of  $t$  and top-p, while the  $y$ -axis signifies CBS. Best viewed on the screen.
5.1 Demographics Analysis
Figure 6 illustrates the relative propor
tions of frequency for all valid demographics. Experiments are conducted on the code generated by InCoder-6B. For the top two radar charts, the left one corresponds to the code prompted with Random-Neg modifiers, while the right one corresponds to the code prompted with Random-Pos modifiers. The arrangement is the same for the bottom two charts. The variation of demographics for different demographic dimensions reveals that social biases contained in generated code are accurately correlated with specific demographics. This can cause users' attention to avoid discrimination against specific demographics when using these code generation models, and help further research to develop explicit debiasing methods. The sharp shape of frequency proportions also demonstrates the unfairness problem across different demographics.
5.2 Effects of Hyper-Parameters
We conduct experiments to study the effects of hyper-parameters of code generation models on the social biases in the code generated by CodeGen-6B. We mainly analyze two hyper-parameters: temperature  $t$  [1] and top-p [14]. Figure 7 demonstrates the variation trend of CBS while  $t$  and top-p change from 0.1 to 0.9. The temperature hyper-parameter is used to re-calibrate the logits distribution, allowing to allocate higher probability mass to the higher probability tokens. We set the values of temperature  $t$  from  $\{0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9\}$ . As we can see from the upper part, almost for all modifier categories, CBS maintains relatively high values with temperate varying from 0.3 to 0.5 and decreases when the temperature is greater than 0.6. Top-p samples tokens from the vocabulary  $(w \in V)$  so that the cumulative probability mass of the sampled tokens exceeds a threshold  $p$ :  $\sum_{w \in V} P(w | w_{1:t-1}) \leq p$ . We set the values of top-p from  $\{0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9\}$ . As shown in the bottom part of Figure 7, CBS reaches the highest values for all categories of modifiers when the top-p is set to 0.8, and remains almost unchanged when the top-p varies from 0.1 to 0.3. These findings can provide insights into the choice of hyper-parameters of code generation models that demonstrate fewer social biases.
Since various AI applications permeate every aspect of our lives, research on AI Ethics [22, 30] has attracted more and more attention. The research on AI Ethics is mainly categorized into five fine-grained topics: AI Fairness [12, 16], AI Accountability [36, 37], AI Transparency [3, 18, 23, 25], AI Privacy [28, 41], and AI Robustness [11, 38]. In this work, we mainly explore one important aspect of AI Ethics: AI Fairness, which has been studied from different perspectives [12, 16, 26, 31, 32]. [24] proposed to study the existence of annotator group bias in various real-world crowdsourcing datasets. [19] measured hierarchical regional bias in pre-trained language models. Some works tried to detect and mitigate social biases in word embeddings [4, 17] and hidden representations [6], while others explored quantifying social biases in downstream tasks. Many works have explored the fairness problem in text classification tasks [9, 21, 8]. Some works also explore the fairness problem in generation tasks, such as machine translation [42], story generation [27], and question answering [35]. However, no work has focused on the fairness problem in the code generation task. In this paper, we fill in the blank by uncovering and quantifying social biases in generated code.
In this paper, we explore the important research topic of code fairness. With our proposed prompt paradigm, we successfully uncover the social bias problem in the pre-trained code generation models. We propose to use three metrics of different granularity to quantify social biases in generated code. Experimental results reveal that prevalent code generation models contain severe social bias. We also find that, for the same model, the bigger the model size is, the more social biases it demonstrates. Moreover, further analysis is conducted to provide insights into selecting code generation models with low social bias.
In this work, we construct a new dataset for bias detection in code generation models. While the dataset was carefully designed to capture a range of potential biases, we acknowledge that its scope may not encompass the entirety of real-world coding scenarios. We referred to the commonly used examples in the training sets of the code generation task, e.g., BIGPYTHON, to design our prompt functions. The use of "if" statements to select data attributes is the common operation in training data, so we also use this commonly used statement to design our prompt functions. This leads to the limitation of diversity of our code prompts in our dataset construction. Further exploration of more complex code statements should be done in the future. Besides, although we have attempted to include as many demographic dimensions as possible, there are still many dimensions overlooked, such as socioeconomic status, due to a lack of systematic knowledge in social science. Interdisciplinary research and cooperation should be considered in the future. Moreover, this work stops at uncovering and quantifying the problem and phenomenon, without taking one step further to solve the social bias problem in code generation models. Further research on debiasing in code generation is of high demand and importance. Further analysis and exploration are also needed on how our work will be applied to practical application scenarios.
[1] David H. Ackley, Geoffrey E. Hinton, and Terrence J. Sejnowski. A learning algorithm for boltzmann machines. Cognitive Science, 9(1):147-169, 1985.
[2] Afra Feyza Akyurek, Sejin Paik, Muhammed Yusuf Kocyigit, Seda Akbiyik, {S}erife Leman Runyun, and Derry Wijaya. On measuring social biases in prompt-based multi-task learning. 2022.
[3] Gloria Andrada, Robert W Clowes, and Paul R Smart. Varieties of transparency: exploring agency within ai systems. AI & society, pages 1-11, 2022.
[4] Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings, 2016.
[5] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidi Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth A. Barnes, Ariel Herbert-Voss, William H. Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Samuel McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. 2021.
[6] Somnath Basu Roy Chowdhury and Snigdha Chaturvedi. Learning fair representations via rate-distortion maximization. arXiv preprint arXiv:2202.00035, 2022.
[7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2018.
[41] Bernd Carsten Stahl and David Wright. Ethics and privacy in ai and big data: Implementing responsible research and innovation. IEEE Security & Privacy, 16(3):26-33, 2018.
[42] Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679-1684, Florence, Italy, July 2019. Association for Computational Linguistics.
[43] Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 12388-12401. Curran Associates, Inc., 2020.
[44] Jun Wang, Benjamin Rubinstein, and Trevor Cohn. Measuring and mitigating name biases in neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, May 2022. Association for Computational Linguistics.
[45] Kellie Webster, Xuezhi Wang, Ian Tenney, Alex Beutel, Emily Pitler, Ellie Pavlick, Jilin Chen, and Slav Petrov. Measuring and reducing gendered correlations in pre-trained models. arXiv: Computation and Language, 2020.