# Program Code Generation with Generative AIs
## 3 Experimental Setup
### Prompt Engineering  
For our tested generative AIs ChatGPT, Bing AI Chat, GH Copilot, StarCoder, Code Llama, CodeWhisperer, and InstructCodeT5+, we performed prompt engineering to find the prompts that led to optimal coding solutions in Java, Python and C++ for the six coding problems mentioned in Section 3.2.2. Figure 2 demonstrates our prompt which worked optimally for instructing ChatGPT to generate Java code (on the right) that solves the LeetCode coding problem (on the left). Our prompt engineering strategy was to stick as close to LeetCode's coding problem description as possible and contained the following steps which are visualized in Figure 3:  
1. We prompted \"Solve the following problem with <programming language> code. <declarations>\" together with the original LeetCode coding problem description, examples, example explanations and constraints. 2. If the result with prompt 1 was _incorrect_ program code (i.e., did not solve the problem) or was _not executable_, we removed the example explanations. 3. If the result with prompt 2 was _incorrect_ program code (i.e., did not solve the problem) or was _not executable_, we removed the examples. 4. If the result with prompt 3 was _incorrect_ program code (i.e., did not solve the problem) or was _not executable_, we took the prompt from prompts 1-3 that resulted in program code that was closest to our human-generated reference. In the next section, we will describe the quality of the program codes which were generated using the optimal prompt from this strategy. \begin{table}
\begin{tabular}{l l} \hline _Halstead Metrics_ & **Calculation** \\ \hline Length of the program (N) & \(N=N_{1}+N_{2}\) \\ Vocabulary of the program (n) & \(n=n_{1}+n_{2}\) \\ Volume of the program (V) & \(V=(N_{1}+N_{2})\log_{2}(n_{1}+n_{2})\) \\ Difficulty of the program (D) & \(D=(n_{1}/2)\cdot(N_{2}/n_{2})\) \\ Programming effort (E) & \(E=D\cdot V\) \\ Implementation time (T) & \(T=E/S\) where \(S=18\) \\ \hline \end{tabular}
\end{table} 
Table 2: Calculation of _Halstead Metrics_. ```
``` Solve the following problem with java code. Class name: Solution, Method name: maxSum(int[]) nums. ```

You are given a 0-indexed integer array nums. You have to find the maximum sum of a pair of numbers from nums such that the maximum digit in both numbers are equal. Return the maximum sum or -1 if no such pair exists. ```
Example 1: Input: runs = [51,71,17,24,42] Output: fill
``` Application: For i = 1 and j = 2, nums[i] and num[j] have equal maximum digits with a pair of 71 + 17 = 81, for i = 3 and j = 4, nums[i] and nums[j] have equal maximum digits with a pair of 24 + 42 = 66. ```
It can be shown that there are no other pairs with equal maximum digits, so the answer is 88. ``` Example 2: Input: runs = [1,2,3,4] Output: -1 Explanation: No pair exists in runs with equal maximum digits. ```
**Containties**
``` Calculate
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
43:
44:
45:
46:
47:
48:
49:
50:
51:
52:
53:
54:
55:
56:
57:
58:
59:
60:
61:
62:
63:
64:
65:
66:
67:
68:
69:
70:
71:
72:
73:
74:
75:
76:
77:
78:
79:
80:
81:
82:
83:
84:
85:
86:
87:
88:
91:
92:
93:
94:
95:
96:
97:
98:
99:
100:
111:
12:
13:
14



## 4 Experiments and Results

To ensure the validity of our experiments, we had all program codes generated by the generative AIs in the short time period from August to September 2023, making sure that for each generative AI only the same version is used. In this section, we will first analyze which generated program codes are _correct_, i.e., solve the coding problems. Then we will compare the quality of the _correct_ program codes using our evaluation criteria _lines of code, cyclomatic complexity_, _time complexity_, _space complexity_, _runtime_, _memory usage_, and _maintainability index_.