# Program Code Generation with Generative AIs
## 2 Related Work  
In this section, we will look at existing work regarding automatic program code generation and evaluation. Codex ([[https://openai.com/blog/openai-codex](https://openai.com/blog/openai-codex)]([https://openai.com/blog/openai-codex](https://openai.com/blog/openai-codex)), accessed on 29 January 2024) is an AI model for program code generation based on GPT-3.0 and fine-tuned on program code publicly available on GitHub [16]. Ref. [16] tested Codex's capabilities in generating Python code using natural language instructions found in in-code comments known as docstrings. They also created _HumanEval_, a dataset of 164 hand-written coding problems in natural language plus their unit tests to assess the _functional correctness_ of program code. One discovery from their research was that if Codex is asked to generate code for the same problem several times, the probability that one generated code is _correct_ increases. Consequently, they used _pass@k_ as an evaluation metric, where \(k\) is the number of generated program codes, and _pass_ is the number of tasks, of which all unit tests were passed. To obtain an unbiased estimation of _pass@k_, ref. [16] applied additional adjustments to the original calculation. Codex achieved a _pass@1_ of 28.8% in solving the provided problems. They compared the Codex-generated code with program code generated by GPT-3.0 and GPT-J [18], but GPT-3.0 demonstrated a _pass@1_ of 0% and GPT-J obtained 11.4%. With _pass@_100, Codex achieved even 70.2%. Ref. [19] evaluated _the validity_, _correctness_, and _efficiency_ of program code generated by GitHub (GH) Copilot using _HumanEval_. GH Copilot is an IDE extension that uses Codex. Ref. [19] defined a code as _valid_, if it was compliant with the syntax rules of a given programming language. The _correctness_ was computed by dividing the programming tasks' passed unit tests by all existing unit tests for this specific task. The _efficiency_ was measured by determining the _time_ and _space complexities_ of the generated codes. Their results demonstrate that Codex was able to generate _valid_ code with a success rate of 91.5%. Regarding code _correctness_, 28.7% were generated _correctly_, 51.2% were generated _partially correctly_, and 20.1% were generated _incorrectly_. Ref. [17] used _HumanEval_ to evaluate the _validity_, _correctness_, _security_, _reliability_, and _maintainability_ of Python code generated by GH Copilot, Amazon CodeWhisperer ([[https://aws.amazon.com/de/codewthisperer](https://aws.amazon.com/de/codewthisperer)]([https://aws.amazon.com/de/codewthisperer](https://aws.amazon.com/de/codewthisperer)), accessed on 29 January 2024), and ChatGPT. They defined a _valid_ code and a _correct_ code as in [19]. To determine the _security_, _reliability_, and _maintainability_, they used SonarQube ([[https://www.sonarqube.org](https://www.sonarqube.org)]([https://www.sonarqube.org](https://www.sonarqube.org)), accessed on 29 January 2024). Their calculation of a code's _security_ is based on the potential cyber-security vulnerabilities of this code. The _reliability_ is based on the number of bugs. The _maintainability_ is measured by counting present code smells. ChatGPT generated 93.3% _valid_ program codes, GH Copilot 91.5%, and CodeWhisperer 90.2%. ChatGPT passed most unit tests by generating 65.2% of _correct_ program codes. GH Copilot reached 46.3%, and CodeWhisperer 31.1%. But when they evaluated newer versions of GH Copilot and CodeWhisperer, they measured 18% and 7% better values for _correctness_. Due to the small number of generated code fragments, the numbers for _security_ were not usable. Concerning _reliability_, ChatGPT produced two bugs, GH Copilot three bugs and CodeWhisperer one bug. CodeWhisperer produced the most _maintainable_ code, ChatGPT the second most, and GH Copilot the least _maintainable_. Ref. [15] introduced a new framework for program code evaluation named _EvalPlus_. Furthermore, they created _HumanEval_+, an extension of _HumanEval_ using _EvalPlus' automatic test input generator_.