# The Impact of Prompting Techniques on the Security of the LLMs and the Systems to Which They Belong
## 1 Introduction  
Large language models are not something completely new that popped up during the COVID-19 pandemic, but the end of 2022 was the moment when these types of artificial neural networks started to gain popularity even among ordinary people that had no prior knowledge of what is a large language model or artificial intelligence, with the release of OpenAI ChatGPT 3. But even before the release of ChatGPT, there were APIs already available that allowed the integration of LLMs into various applications. There is no doubt the \"commercial\" success of ChatGPT made the adoption of AI technologies and LLMs in particular grow. But it did not take long until researchers found that there are ways in which they can improve LLMs performance by using prompt engineering techniques [1; 2; 3; 4]. These types of techniques enhance the abilities of the large language models, mitigating their weaknesses and allowing them to interact with other software components. However, it seems that these new tools come with a large spectrum of security vulnerabilities, such as prompt injection and jailbreak [5; 6], and all their variations. To better understand where this set of vulnerabilities comes from, we look into the the mechanisms of four of the most famous prompting techniques and try to create a causality relationship between the characteristics of the prompting techniques and possible vulnerabilities. To have a better understanding of what the challenges are when deploying an LLM into a production environment, we try to learn from others' experiences and look into a case study in which some OpenAI researchers try to create an LLM-based solution to automatic content moderation. Utilizing all these insights, we conduct an analysis on the two main popular attacks: prompt injection and jailbreak. By refining the two terms, we come up with a clear distinction between them that allows us to individually testthe reaction of LLM-based components that are using specific prompting techniques for different kinds of attacks that we implemented using different paradigms proposed in previous work [5; 6; 7; 8]. The results of our test validate the presence of multiple security issues with the implemented attacks having in most of the cases a 50% success rate. The last part of this paper consists of a discussion around possible defense strategies that could be used against the presented attacks as well as their possible advantages and disadvantages. In conclusion, our work makes the following contributions:  
* A different perspective regarding prompting techniques. So far, most of the papers that tackle the prompt engineering subject do not take into consideration their security implication. * Refined definitions for prompt injection and jailbreak. These two terms seemed to be often confused one with another. We set a clear distinction between the two by the intention of the attacker and the result of the attack. * A set of test cases to test for different types of vulnerabilities. Our set of attacks and scenarios represents a good starting point for a possible security benchmark library that could be used to audit LLM-based system components. This manuscript is organized into several key sections: The Introduction provides a comprehensive overview of large language models, outlining their growth and current significance, along with the motivation behind this research. The Domain Overview section delves into the foundational principles and prior work that underpin large language models and prompt engineering. The Related Work section surveys the existing literature, highlighting recent advancements in the field, including various techniques to enhance large language models' performance and security. In the Technology Stack section, we discuss the tools and frameworks used for conducting our experiments, including the OpenAI API and Langchain. The subsequent Attacks section examines vulnerabilities and presents an analysis of security risks through practical experiments, focusing on prompt injection and jailbreak attacks.