A Simple, Yet Effective Approach to Finding Biases in Code Generation
Year: 2023

Abstract:
Recently, high-performing code generation systems based on large language models have surfaced. They are trained on massive corpora containing much more natural text than actual executable computer code. This work shows that current code generation systems exhibit undesired biases inherited from their large language model backbones, which can reduce the quality of the generated code under specific circumstances. To investigate the effect, we propose the ”block of influence” concept, which enables a modular decomposition and analysis of the coding challenges. We introduce an automated intervention mechanism reminiscent of adversarial testing that exposes undesired biases through the failure modes of the models under test. Finally, we demonstrate how our framework can be used as a data transformation technique during fine-tuning, acting as a mitigation strategy for these biases.

heading: Future Work
text: 1, we present how an adaptation to less formatted codebases would be possible, but for now, we leave it as a future investigation. Finally, there is no guarantee that the improved performance against the suggested perturbations reflects an equivalent performance increase in realworld code assistant applications. Real-time coding suggestions and completions that are more user aligned are out of the scope of this work. 9 Risks and Ethical Considerations Our research aims to discover and remove biases in code-generation scenarios through adversarial intervention. However, we acknowledge that insecure or malicious code can still be generated after finetuning with our suggested augmentations. Furthermore, our work is focused only on cognitive biases that affect the reasoning and logic behind the coding process of large language models. Social biases and stereotypes can still appear when general-purpose LLMs such as Codex or Bloom are used in typical text generation scenarios. Signs of robustness against our methods are not to be confused with indicators of other forms of biases not existent