# Program Code Generation with Generative AIs
###### Abstract  
Our paper compares the _correctness, efficiency_, and _maintainability_ of human-generated and AI-generated program code. For that, we analyzed the computational resources of AI- and human-generated program code using metrics such as _time_ and _space complexity_ as well as _runtime_ and _memory usage_. Additionally, we evaluated the _maintainability_ using metrics such as _lines of code, cyclomatic complexity_, _Halstead complexity_ and _maintainability index_. For our experiments, we had generative AIs produce program code in Java, Python, and C++ that solves problems defined on the competition coding website _letcode.com_. We selected six LeetCode problems of varying difficulty, resulting in 18 program codes generated by each generative AI. GitHub Copilot, powered by Codex (GPT-3.0), performed best, solving 9 of the 18 problems (50.0%), whereas CodeWhisperer did not solve a single problem. BingAI Chat (GPT-4.0) generated _correct_ program code for seven problems (38.9%), ChatGPT (GPT-3.5) and Code Llama (Llama 2) for four problems (22.2%) and StarCoder and InstructCodeT5+ for only one problem (5.6%). Surprisingly, although ChatGPT generated only four _correct_ program codes, it was the only generative AI capable of providing a _correct_ solution to a coding problem of difficulty level _hard_. In summary, 26 AI-generated codes (20.6%) solve the respective problem. For 11 AI-generated _incorrect_ codes (8.7%), only minimal modifications to the program code are necessary to solve the problem, which results in time savings between 8.9% and even 71.3% in comparison to programming the program code from scratch.  
artificial intelligence; generative AIs; AI program code generation; program code efficiency; program code maintainability +
Footnote †: journal:  
definition  
they may not always produce _correct_, _efficient_, or _maintainable_ program codes. Therefore, it is crucial to carefully consider various aspects when deciding whether or not to use these tools [14].  
While substantial research has focused on evaluation metrics for _correctness_ like _pass@k_[15; 16; 17], there is a noticeable gap: The extensive comparison of AI-generated and human-generated program codes based on various metrics has largely been understudied. Consequently, our study embarks on a comprehensive and in-depth exploration of the coding capabilities of seven state-of-the-art generative AIs. Our goal is to evaluate the AI-generated program codes based on the interaction of various metrics such as _cyclomatic complexity_, _maintainability index_, etc., concerning their _correctness_, _efficiency_ and _maintainability_. Moreover, we go one step further by comparing the AI-generated codes with corresponding human-generated codes written by professional programmers.  
Our contributions are as follows:  
* We investigate and compare the _correctness_, _efficiency_ and _maintainability_ of AI-generated program codes using varying evaluation metrics.
* We are the first to extensively compare AI-generated program codes to human-generated program codes.
* We analyze the program codes that address problems of three difficulty levels--_easy_, _medium_ and _hard_.
* We produced a dataset of 126 AI-generated and 18 human-generated program codes--the _AI/Human-Generated Program Code Dataset_--which we share with the research community ([[https://github.com/Back3474/AI-Human-Generated-Program-Code-Dataset](https://github.com/Back3474/AI-Human-Generated-Program-Code-Dataset)]([https://github.com/Back3474/AI-Human-Generated-Program-Code-Dataset](https://github.com/Back3474/AI-Human-Generated-Program-Code-Dataset)), accessed on 29 January 2024).  
In the next section, we will provide an overview of related work. In Section 3, we will present our experimental setup. Our experiments and results will be described in Section 4. In Section 5, we will conclude our work and indicate possible future steps.