# An Empirical Study of the Code Generation of Safety-Critical Software Using LLMs
## 2 Backgrounds and Motivation
### Prompt Engineering for Code LLMs  
From the perspective of applying large language models, prompts are the sole input for these models. It is evident that in order to achieve good generation results, the most crucial aspect is to provide the most appropriate prompts for the large language model, which is known as prompt engineering. The quality of the output(s) generated by conversational LLMs is directly related to the quality of the prompts provided by the user [29]. In 2023, OpenAI released dedicated guidelines for prompt engineering. The importance and validation of good prompt design with LLMs is well proven [30; 31; 32; 33; 34; 35]. Specifically, in the context of code generation based on large language models, [PERSON] et al., [PERSON] et al., [PERSON] et al., [PERSON] et al., [PERSON] et al., and [PERSON] et al. have proposed prompt methods for code generation [7; 8; 26; 27; 28; 36]. These methods primarily generate prompts from software functional descriptions, code language constraints, detailed step descriptions, and functional examples, among other aspects, as shown in Table 1. These prompt methods have achieved certain effectiveness in testing various large language models, mainly targeting general program functionalities and sub-algorithms. However, no exploration has been conducted regarding prompt engineering for top-level functional software in the industrial domain. In addition, these research studies mainly focus on Python, Java, and other high-level languages, but rarely on the generation of C language in the embedded system.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Reference** & **Prompt Examples** & **Feature** \\ \hline [PERSON] et al., 2023 [27] & Prompt1: “Write a Python function to tell me what the date is today”; & function descriptions and code \\  & Prompt2: “List the prime numbers between 20 and 30 with Java”. & language constraints \\ \hline [PERSON] et al., 2021 [26] & Prompt1: “Write a python function to check if a given number is one less than twice its reverse” & function descriptions and code \\ \hline  & Prompt1: “Write a function vowels\_count which takes a string & language constraints \\  & representing a word as input and returns the number of vowels in the string. & \\  & Vowels in this case are ‘a’, ‘e’, ‘i’, ‘o’, ‘u’. & \\ [PERSON] et al., 2021 [8] & Here, ’y’ is also a vowel, but only when it is at the end of the given word. & software function descriptions \\  & Example: & and functional examples \\  & >>> vowels\_count(“abcde”) & \\  & 2 & \\  & >>> vowels\_count(“ACEDY”) & \\  & 3” & \\ \hline  & Prompt1: “How to convert a numpy array of dtype=object to torch & \\  & Tensor? & \\  & array([ & function descriptions \\  & array([ & function descriptions \\  & array([ & \\  & dtype=object)” & \\ \hline  & Prompt1: & \\  & “Can you write a 3D game in HTML with Javascript, I want: & \\  & -There are three avatars, each is a sphere. & \\  & -The player controls its avatar using arrow keys to move. & \\  & -The enemy avatar is trying to catch the player. & software function descriptions, \\ [PERSON] et al., 2023 & -The defender avatar is trying to block the enemy. & code language constraints, \\  & -There are also random obstacles as cubes spawned randomly at the & detailed step descriptions \\  & beginning and moving randomly. The avatars cannot cross those cubes. & \\  & -The player moves on a 2D plane surrounded by walls that he cannot & \\  & cross. The wall should cover the boundary of the entire plane. & \\  & -Add physics to the environment using cannon”. & \\ \hline \hline \end{tabular}
\end{table}
Table 1: Summary of prompts of code LLMs.