# The Impact of Prompting Techniques on the Security of the LLMs and the Systems to Which They Belong
## 4 Attacks
### Jailbreak vs. Prompt Injection  
Jailbreak and prompt injection are two of the most popular kinds of attacks that we see mentioned when the problem of the LLMs security is discussed. Often, these two terms are used as synonyms for a kind of attack in which the user manipulates the LLM to perform an undesirable action. We must mention that in this context, action does not refer to an interaction with an outside environment as in ReAct [4] but rather any kind of response from the LLM to the prompt of the user. We want to better define these two termsbecause even if they have similar working mechanisms behind them, we think they target two different kinds of LLM-based components. We differentiate between the two by the intention of the attacker and the result of the attack. We take the same approach as the authors of \"Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study\" [6] and start from the meaning of jailbreak. In the technology context, jailbreaking means to reverse engineer a system and exploit its vulnerabilities in order to have full control and privileges over the system. In this manner, we define a jailbreak attack as a vicious prompt which has an objective to bypass of any kind safety mechanism or ethical guidelines. This type of attack has as the end target the LLM itself. We associate this attack usually with chatbots like ChatGPT. The attack is successful if after the attack prompt, the chatbot does not follow the ethical guidelines that were meant and breaks all restrictions. It is important to note that in this case, the LLM serves the same initial scope only without any kind of boundaries. The following example is meant to clarify the definition: a malicious user of ChatGPT uses the DAN (Dan-Do anything now) [32] prompt and jailbreaks ChatGPT. After that, the users ask ChatGPT to write racist jokes. In this context, the user's intent is to attack the ChatGPT and not another system component of OpenAI in general, so the end target is indeed the chatbot. Then, ChatGPT is used to create unethical jokes but is doing so by following its initial instruction of assisting the user but without taking into consideration the ethical guidelines it was programmed to follow; see Figure 9.  
The term prompt injection has its roots in the name of one the most \"famous\" vulnerabilities that plagued the Web, SQL injection; see Figure 10. SQL injection is an attack where SQL statements are inserted into an entry field of an application, allowing attackers to manipulate the database, retrieve sensitive data, bypass authentication, and potentially compromise the entire system. It exploits vulnerabilities in input validation and query handling. Keeping the same mechanism, we define prompt injection as a vicious prompt that has as its objective to influence the output generated by the LLM, making it perform another task than the one it was supposed to or manipulating the model in generating vicious instruction that is passed further in the system. We observe that in this case, the end target is not necessarily the LLM. The LLM can act as a bridge that allows the attacker to  
Figure 9: Normal flow vs. jailbreak flow. damage other components that otherwise would not be accessible. The attack is successful if after the attack prompt, the scope of the model has changed and it performed the injected task instead. The impact of the attack depends on the level of the targeted component. If the attack wants to hijack the scope of the LLM, then it is less serious than if it uses the LLM to conduct an RCE or SQL injection attack. Another important aspect is the fact that when talking about prompt injection, the prompt must not come directly from a user all the time. A malicious prompt could be read by the model from a database where it was planted or from other places when solving another task. An example in this sense would be an LLM that must process all incoming emails and sort them based on their content.