# Program Code Generation with Generative AIs
## 3 Experimental Setup
### Prompt Engineering
In the _medium_ task#3, where only _[PERSON]_ produced _correct_ program code in Python (_P_), _[PERSON]_ was even able to solve the problem with a _cyclomatic complexity_ of 3, which is 67% less than _human_. _[PERSON]_ generated code with 50% less _cyclomatic complexity_ to solve task#6 in Python (_P_). _[PERSON]_ and _[PERSON]_ produced code with 33% less _cyclomatic complexity_ to solve task#3 in Java (_J_). _[PERSON]_ and _[PERSON]_ also generated code with 33% lower _cyclomatic complexity_ to solve task#3 in C++ (C). Moreover, 25% less _cyclomatic complexity_ is in the C++ code (C) for task#4. \begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & \multicolumn{2}{c}{**Easy**} & \multicolumn{2}{c}{**Medium**} & \multicolumn{2}{c}{**Hard**} \\  & **\#1** & **\#2** & **\#3** & **\#4** & **\#5** & **\#6** \\  & **J** | **P** | **C** & **J** | **P** | **C** & **J** | **P** | **C** & **J** | **P** | **C** & **J** | **P** | **C** & **J** | **P** | **C** \\ \hline ChatGPT (CPT-3.5) &

### Time Complexity

Table 6 illustrates the _time complexity_ in the 26 _correct_ program codes plus the _time complexity_ in our human-written reference program codes (_human_). The _time complexity_ quantifies the upper bound of time needed by an algorithm as a function of the input [56] as described in Section 3.3.3. The lower the order of the function, the better the _complexity_. The cross-column entries mean that the value is the same in all cross-columns. Out of the 26 _correct_ program codes, in one case (4%) (_AI_) a generative AI was able to solve the coding problem with code that has less _time complexity_ than _human_. Thirteen AI-generated program codes (50%) were outperformed by _human_ in this evaluation metric. Twelve program codes (46%) contain the same _time complexity_ as _human_. _GH Copilot_ performed better than the other generative AIs being able to generate one program code with lower _time complexity_ than _human_ and four program codes with equal _time complexity_. _ChatGPT_, _Bing AI_, and _Code Llama_ produced program code with the same _time complexity_ as _human_ in two program codes each. ### Space Complexity

Table 7 illustrates the _space complexity_ in the 26 _correct_ program codes plus the _space complexity_ in our human-written reference program codes (_human_). Similar to the _time complexity_, the _space complexity_ quantifies the upper bound of space needed by an algorithm as a function of the input [57] as described in Section 3.3.3. Analogous to the _time complexity_, the lower the order of the function, the better the _complexity_. The cross-column entries also mean that the value is the same in all cross-columns. Out of the 26 _correct_ program codes, in seven cases (27%) (_AI_) a generative AI was able to solve the coding problem with code that has less _space complexity_ than _human_. One AI-generated program code (4%) was outperformed by _human_ in this evaluation metric. Eighteen program codes (69%) contain the same _space complexity_ as _human_. This shows that the generative AIs perform significantly better in terms of _space complexity_ compared to _time complexity_. Again, _GH Copilot_ outperformed the other generative AIs being able to generate three program codes with lower _space complexity_ than _human_ and five program codes with equal _space complexity_. _Bing AI_ generated program code with the same _space complexity_ as _human_ in five program codes, _ChatGPT_ and _Code Llama_ in three program codes each, as well as _StarCoder_ and _InstructCodeT5+_ in one program code each. ### Runtime

Table 8 demonstrates the _runtime_ of the 26 _correct_ program codes plus the _runtime_ of our human-written reference program codes (_human_) on LeetCode in milliseconds. The lower the _runtime_ of a program code, the better. The _runtime_ of the six _correct_ program codes labeled with \"*\" could not be measured since their execution resulted in a _Time Limit Exceeded_ error when executed on LeetCode.