# An Empirical Study of the Code Generation of Safety-Critical Software Using LLMs
## 2 Backgrounds and Motivation  
### Generative Pre-Training Large Language Models  
In 2017, [PERSON] et al. from Google proposed the Transformer deep learning model architecture, which laid the foundation for the current mainstream algorithm architecture in large models [10]. In 2018, Google introduced the large-scale pre-trained language model BERT (bidirectional encoder representations from transformers) based on the Transformer framework [11]. This model is a bidirectional deep pre-trained model based on Transformer, and its model parameters exceeded 300 million for the first time (BERT-Large has approximately 340 million parameters). In the same year, OpenAI introduced the generative pre-training Transformer model, GPT (generative pre-training) [12], demonstrating the potential of large models in natural language processing and driving the development of the natural language processing field. In 2020, OpenAI released the GPT-3 model [13] with approximately 175 billion parameters. GPT-3 demonstrated capabilities surpassing human-level performance in multiple natural language processing (NLP) tasks. In January 2021, Google Brain proposed the Switch Transformer model [14], becoming the first trillion-parameter language model in history, with a parameter count as high as 1.6 trillion. In December of the same year, Google also introduced GLaM [15], a general sparse model with 120 million parameters, which outperformed GPT-3 in multiple few-shot learning tasks. In March 2022, OpenAI unveiled InstructGPT (175B), which utilized instruction learning and reinforcement learning from human feedback (RLHF) to guide the model's training [16]. Using a similar model structure and training approach, and trained on a large scale dialogue dataset, OpenAI released ChatGPT on 30 November 2022 [17]. The language understanding ability and content generation effectiveness of ChatGPT were astonishing, and within just two months, the number of monthly active users of ChatGPT exceeded 100 million. ChatGPT marked a milestone for large language models, truly showcasing the powerful capabilities of pre-trained large models and making people fully aware of the enormous potential of pre-trained large language models. On 14 March 2023, OpenAI released a new milestone for pre-trained large language models: the more powerful GPT-4. GPT-4 not only significantly improved answer accuracy but also possessed higher-level image recognition capabilities and the ability to generate lyrics and creative text with style variations [18]. On 10 May 2023, Google officially released the new general-purpose large language model PaLM2, which supports mathematical operations, software development, language translation inference, and natural language generation [19].On 13 July 2023, Anthropic announced the release of its latest artificial intelligence model, Claude 2, which allows inputs of up to 100,000 tokens per prompt (equivalent to approximately 750,000 English words) and generates longer documents, capable of writing thousands of tokens at once (equivalent to several thousand words or characters). Claude 2 achieved this breakthrough in terms of input and output token lengths [20]. On 19 July 2023, Meta released the open-source large language model Llama2, becoming the first freely available open-source large language model for commercial use [21].