# Program Code Generation with Generative AIs
## 3 Experimental Setup
### Prompt Engineering
\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & & & & \multicolumn{4}{c}{**Estimated Time To**} \\  & **\#** & **Lang.** & **MI** & \(T_{incorrect}\) \({}^{1}\) \(T_{correct}\) & **TTC** & \(\Delta\)\(T_{correct}\)-TTC (\%) \\ \hline StarCoder & 1 & C & 49.77 & 1001 \(\mid\) 1225 & 1234 & \(-\)0.73 \\ CodeWhiperer & 1 & J & 57.55 & 713 \(\mid\) 1167 & 1693 & \(-\)31.07 \\ StarCoder & 1 & J & 57.78 & 827 \(\mid\) 1464 & **1241** & **+17.97** \\ ChatGPT (GPT-3.5) & 3 & J & 49.73 & 1570 \(\mid\) 2984 & 3001 & \(-\)0.57 \\ StarCoder & 3 & P & 57.41 & 1045 \(\mid\) 450 & 1370 & \(-\)67.15 \\ CodeWhiperer & 2 & C & 58.46 & 355 \(\mid\) 361 & **258** & **+39.92** \\ ChatGPT (GPT-3.5) & 3 & P & 60.48 & 303 \(\mid\) 865 & **760** & **+13.82** \\ ChatGPT (GPT-3.5) & 2 & J & 52.97 & 656 \(\mid\) 500 & 738 & \(-\)32.25 \\ Code Llama (Llama 2) & 1 & P & 64.87 & 492 \(\mid\) 406 & **352** & **+15.34** \\ Bing AI Chat (GPT-4.0) & 3 & J & 49.05 & 1910 \(\mid\) 1846 & 2048 & \(-\)9.86 \\ InstructCodefT5+ & 3 & C & 56.69 & 757 \(\mid\) 829 & **650** & **+27.54** \\ Code Llama (Llama 2) & 4 & J & 53.78 & 538 \(\mid\) 574 & **498** & **+15.26** \\ InstructCodefT5+ & 2 & C & 64.83 & 187 \(\mid\) 356 & **279** & **+27.60** \\ StarCoder & 2 & P & 63.38 & 254 \(\mid\) 209 & **192** & **+8.85** \\ ChatGPT (GPT-3.5) & 1 & P & 56.59 & 697 \(\mid\) 577 & 655 & \(-\)11.91 \\ CodeWhiperer & 2 & J & 57.89 & 459 \(\mid\) 355 & 438 & \(-\)18.95 \\ CodeLlama (Llama 2) & 3 & P & 63.98 & 364 \(\mid\) 275 & 294 & \(-\)6.46 \\ StarCoder & 2 & C & 57.45 & 458 \(\mid\) 395 & 402 & \(-\)1.74 \\ InstructCodefT5+ & 2 & J & 56.99 & 519 \(\mid\) 365 & 546 & \(-\)33.15 \\ StarCoder & 1 & P & 57.89 & 682 \(\mid\) 628 & **550** & **+14.18** \\ Bing AI Chat (GPT-4.0) & 4 & J & 49.22 & 648 \(\mid\) 552 & 765 & \(-\)27.84 \\ Bing AI Chat (GPT-4.0) & 4 & P & 57.16 & 450 \(\mid\) 436 & 454 & \(-\)3.96 \\ CodeWhiperer & 2 & P & 63.55 & 204 \(\mid\) 209 & **122** & **+71.31** \\ CodeWhiperer & 3 & J & 56.93 & 605 \(\mid\) 752 & **605** & **+24.30** \\ \hline \hline \end{tabular}
\end{table} 
Table 11: Potential of _Incorrect_ AI-Generated Program Code. ## 5 Conclusions and Future Work

The fast development of AI raises questions about the impact on developers and development tools since with the help of generative AI, program code can be generated automatically. Consequently, the goal of our paper was to answer the question: How efficient is the program code in terms of computational resources? How understandable and maintainable is the program code for humans? To answer those questions, we analyzed the computational resources of AI- and human-generated program code using metrics such as _time_ and _space complexity_ as well as _runtime_ and _memory usage_. Additionally, we evaluated the _maintainability_ using metrics such as _lines of code_, _cyclomatic complexity_, _Halstead complexity_ and _maintainability index_. In our experiments, we utilized generative AIs, including ChatGPT (GPT-3.5), Bing AI Chat (GPT-4.0), GH Copilot (GPT-3.0), StarCoder (StarCoderBase), Code Llama (Llama 2), CodeWhisperer, and InstructCodeT5+ (CodeT5+) to generate program code in Java, Python, and C++. The generated program code aimed to solve problems specified on the coding platform _leetcode.com_. We chose six LeetCode problems with varying difficulty, resulting in the generation of 18 program codes. GH Copilot outperformed others by solving 9 out of 18 coding problems (50.0%), while CodeWhisperer failed to solve any coding problem. BingAI Chat provided _correct_ program code for seven coding problems (38.9%), while ChatGPT and Code Llama were successful in four coding problems (22.2%). StarCoder and InstructCodeT5+ each solved only one coding problem (5.6%). GH Copilot excelled in addressing our Java and C++ coding problems, while BingAI demonstrated superior performance in resolving Python coding problems.