# Program Code Generation with Generative AIs
## 3 Experimental Setup
### Prompt Engineering
\begin{table}
\begin{tabular}{l c



Out of the 26 _correct_ program codes, in six cases (23%) a generative AI was able to solve the coding problem with code that executes with less _runtime_ than _human_. Seventeen AI-generated program codes (65%) were outperformed by _human_ in this evaluation metric. Three program codes (12%) took the same _runtime_ as _human_. _[PERSON]_ performed better than the other generative AIs being able to generate two program codes with less _runtime_ than _human_ and one program code with the same _runtime_. In easy task#1, where only _GH [PERSON]_ produced the _correct_ program code in Java (_J_), _[PERSON]_ was even able to solve the problem in a _runtime_ of 4 milliseconds, which is 125% less than _human_. _InstructCodeT5_+ generated code that took 55% less _runtime_ to solve task#2 in Python (_P_). _[PERSON]_ produced code that took 12% less _runtime_ to solve task#4 in C++ (C). _[PERSON]_, _[PERSON]_ and _[PERSON]_ generated code that took the same _runtime_ as _human_ to solve task#2 in Java (_J_). ### Memory Usage

Table 9 lists the _memory usage_ of the 26 _correct_ program codes plus the _memory usage_ of our human-written reference program codes (_human_) on LeetCode in megabytes. The lower the _memory usage_ of a program code, the better. The _memory usage_ of the six _correct_ program codes labeled with \"*\" could not be measured since their execution resulted in a _Time Limit Exceeded_ error when executed on LeetCode. Out of the 26 _correct_ program codes, in five cases (19%) a generative AI was able to solve the coding problem with code that executes with less _memory usage_ than _human_. Eleven AI-generated program codes (42%) were outperformed by _human_ in this evaluation metric. Ten program codes (38%) took the same _memory usage_ as _human_. _GH Copilot_ performed better than the other generative AIs being able to generate one program code with less _memory usage_ than _human_ and five program codes with the same _memory usage_, closely followed by _[PERSON]_ which was able to generate one program code with less _memory usage_ than _human_ and three program codes with the same _memory usage_. However, in the AI-generated program codes, which have lower _memory usage_, the _memory usage_ of less than 1% relative is not significantly lower than in _human_. \begin{table}
\begin{tabular}{l c c c c} \hline \hline  & \multicolumn{2}{c}{**Easy**} & \multicolumn{2}{c}{**Medium**} & \multicolumn{2}{c}{**Hard**} \\  & **\#1** & **\#2** & **\#3** & **\#4** & **\#5** & **\#6** \\  & **J** | **P** | **C** & **J** | **P** | **C** & **J** | **P** | **C** & **J** | **P** | **C** & **J** | **P** | **C** \\ \hline ChatGPT (CPT-



### Maintainability Index

Table 10 demonstrates the _maintainability index_ of the 26 _correct_ program codes plus the _maintainability index_ of our human-written reference program codes (_human_). The higher the _maintainability index_ of a program code, the better. Out of the 26 _correct_ program codes, in 13 cases (50%) a generative AI was able to produce code with a higher _maintainability index_ than _human_. Thirteen AI-generated program codes (50%) were exceeded by _human_ in this evaluation metric. _GH Copilot_ and _Bing AI_ performed better than the other generative AIs being able to generate four program codes (15%) with a higher _maintainability index_ than _human_. _ChatGPT_ and _Code Llama_ produced two program codes (8%) with a higher _maintainability index_ than _human_. ### Potential of Incorrect AI-Generated Program Code

After analyzing the 26 _correct_ program codes, our goal was to evaluate which of the 96 _incorrect_ and the four _not executable_ program codes have the potential to be easily modified manually and then used quickly and without much effort to solve the corresponding coding problems. Consequently, we first had programmers determine which of the 96 _incorrect_ program codes have the potential to be corrected manually due to their _maintainability_ and proximity to the _correct_ program code.