# A CIA Triad-Based Taxonomy of Prompt Attacks on Large Language Models
## 3 Taxonomy of Prompt Attacks
### Mathematical Representations of Prompt Attacks on LLMs  
Prompt attacks on LLMs exploit their ability to generate outputs based on crafted inputs, often resulting in undesired or malicious outcomes. These attacks target the LLM's Confidentiality, Integrity, and Availability (CIA) by leveraging adversarial prompts. To understand and mitigate these vulnerabilities, it is essential to examine the mathematical foundations underpinning such attacks.  
An LLM can be represented as a function \(f\) that maps a prompt \(p\) from the prompt space \(\mathcal{P}\) to an output \(o\) in the output space \(\mathcal{O}\):  
\[o=f(p).\]  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline  
Adversarial prompts \(p_{a}\in\mathcal{P}\) are specifically designed to manipulate the model, producing malicious outputs \(o_{a}\) that deviate from the intended behavior:  
\[o_{a}=f(p_{a}).\]  
The mathematical representations of prompt attacks involve the following parameters:  
1. _Prompt Space and Outputs_: * \(p\): The input prompt provided to the LLM. * \(o=f(p)\): The output generated by the LLM for a given prompt \(p\). * \(p_{a}\): An adversarially-crafted prompt designed to produce malicious or undesired outputs.
2. _Perturbations and Bias_: * \(\delta p\): A perturbation or modification added to \(p_{\text{original}}\), often representing malicious instructions. * \(\delta b\): A bias introduced into the prompt that affects fairness or neutrality.
3. _Likelihood and Similarity_: * \(S(f(p),D)\): A similarity function that measures how closely the model's output \(f(p)\) matches sensitive data \(D\). * \(P(x|f(p))\): The likelihood of sensitive data \(x\) being inferred based on the model's output \(f(p)\).
4. _Context and Token Limits_: * \(\mathcal{C}\): The context window, which includes a sequence of prompts. * \(L_{\text{context}}\): The maximum token limit for the model's input.
5. _Other Functions_: * \(g(p,\cdot)\): A transformation function that modifies \(p\), such as injecting bias or semantic drift. * \(\epsilon\sim\mathcal{N}(0,\sigma^{2})\): A small perturbation added to a prompt that is used in adversarial example attacks.  
Table 3 categorizes different types of prompt attacks, presenting their mathematical formulations alongside detailed explanations.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Attack Type** & **Mathematical Representation** & **Explanation** \\ \hline \multirow{4}{*}{**Data Extraction Attack**} & \multirow{4}{*}{\(p_{a}=\operatorname*{argmax}_{p\in\mathcal{P}}S(f(p),D)\)} & Measures the similarity \(S\) between the model’s output \(f(p)\) and sensitive data \(D\). The attacker crafts prompts to maximize similarity and retrieve sensitive data. \\ \hline \multirow{4}{*}{**Instruction Injection**} & \multirow{4}{*}{\(f(p)=f(p_{\text{original}})+\delta p,\quad\delta p\sim\) Malicious Instructions} & The attacker appends malicious instructions \(\delta p\) to the original prompt \(p_{\text{original}}\), altering the model’s intended behavior. \\ \hline \multirow{2}{*}{**Toxic Prompting**} & \multirow{2}{*}{\(o_{\text{toxic}}=f(p_{\text{toxic}}),\quad p_{\text{toxic}}=g(p,\text{bias})\)} & The attacker introduces biases into the crafted prompt \(p_{\text{toxic}}\) to generate harmful or offensive content. \\ \cline{1-1} \cline{2-3}  & & & \\ \hline \hline \end{tabular}
\end{table}
Table 3: Mathematical representations of prompt attacks on LLMs.