# Program Code Generation with Generative AIs
## 3 Experimental Setup
### Prompt Engineering
Finally, we will evaluate which of the _incorrect_ program codes--due to their _maintainability_ and proximity to the _correct_ program code--have the potential to be easily modified manually and then used quickly and without much effort to solve the corresponding problems. ### Correct Solutions

Table 3 demonstrates which generated program codes for our six tasks in the three programming languages Java (_J_), Python (_P_) and C++ (_C_) were _correct_ (indicated with ), i.e., did solve the coding problem. The entry \"--\" indicates program code that was _incorrect_, i.e., did not solve the coding problem. We observe that 122 of the 126 AI-generated program codes are _executable_. Four program codes result in a compilation error which is indicated with. Out of our 18 programming tasks, _GH Copilot_ was able to generate nine _correct_ program codes (50%) in total (), followed by _Bing AI_ with seven _correct_ program codes (39%). _ChatGPT_ and _Code Llama_ produced only four _correct_ program codes (22%). _CodeWhisperer_ did not produce _correct_ program code in all three programming languages. This results in _26 correct_ program codes, which we will further analyze in Sections 4.2-4.8. Looking at the difficulty levels shows that the generative AIs rather generated _correct_ program code for _easy_ and _medium_ coding problems. Regarding the programming languages, most _correct_ Java (_J_) and C++ (_C_) code were generated with _GH Copilot_. Most _correct_ Python (_P_) code was produced with _Bing AI_. The program code which was written by human programmers (_human_) was always _correct_, independent of the programming language and the difficulty of the coding problem. ### Lines of Code

Table 4 shows the number of code lines in the _correct_ 26 program codes which solve the corresponding coding problem plus the number of code lines in our human-written reference program codes (_human_). Out of the 26 _correct_ program codes, in eight cases (31%) a generative AI was able to solve the coding problem with code that contains fewer _lines of code_ than _human_. However, 13 AI-generated program codes (50%) were outperformed by _human_ in this evaluation metric. For task#2, all five corresponding program codes (19%) contained the same number of _lines of code_ as _human_. _[PERSON]_ and _GH Copilot_ performed better than the other generative AIs, being able to generate three program codes with fewer _lines of code_ than _human_. \begin{table}
\begin{tabular}{l



In the _hard_ task#6, where only _[PERSON]_ produced a _correct_ program code, _[PERSON]_ was even able to solve the problem with 10 Python (_P_) _lines of code_, which is 60% fewer _lines of code_ than _human_. _[PERSON]_ generated 86% fewer _lines of code_ to solve task#3 in Python (_P_). _[PERSON]_ and _GH Copilot_ produced 36% fewer _lines of code_ to solve task#2 in Java (_J_); 22% fewer C++ (_C_) _lines of code_ are required in _[PERSON]_'s program code for task#3. Furthermore, 10% fewer Java (_J_) _lines of code_ are required in _[PERSON]_'s and _[PERSON]_'s program code for task#3. ### Cyclomatic Complexity

Table 5 shows the _cyclomatic complexity_ in the 26 _correct_ program codes which solve the coding problem plus the _cyclomatic complexity_ in our human-written reference program codes (_human_). The _cyclomatic complexity_ measures the number of linearly independent paths through a program code. As described in Section 3.3.2, a higher _cyclomatic complexity_ indicates worse code quality. Out of the 26 _correct_ program codes, in seven cases (27%) a generative AI was able to solve the coding problem with program code that has less _cyclomatic complexity_ than _human_. Only four AI-generated program codes (15%) were outperformed by _human_ in this evaluation metric. Fifteen program codes (58%) contain the same _cyclomatic complexity_ as _human_. _[PERSON]_ performed better than the other generative AIs being able to generate three program codes with less _cyclomatic complexity_ than _human_.