# Neighborhood Deviation Attack Against In-Context Learning
## 3 Preliminaries  
### Large Language Model  
A Language Model (LM) is a type of machine learning model designed to model the probability distribution of natural language and estimate the probability of any linguistic sequence. In recent years, with the rapid development of deep learning algorithms and hardware computational power, the number of parameters in LMs has continuously in \begin{table}
\begin{tabular}{l c c c c c} \hline \hline \multirow{2}{*}{**Attack Method**} & \multirow{2}{*}{**Target**} & \multirow{2}{*}{**Attack Content**} & \multicolumn{2}{c}{**Training Data**} & \multicolumn{2}{c}{**Target Model**} \\ \cline{3-6}  & & & **Sample** & **Distribution** & **Structure** & **Parameter** \\ \hline Shadow Attack [19] & ML Model & Training Data & ✓ & ✓ & ✓ & \(\times\) \\ Sampling Attacks [20] & ML Model & Training Data & \(\times\) & ✓ & ✓ & \(\times\) \\ LOGAN [21] & Generative Model & Training Data & \(\times\) & \(\times\) & ✓ & ✓ \\ [PERSON] et al. [22] & ML Model & Training Data & \(\times\) & \(\times\) & ✓ & \(\times\) \\ [PERSON] et al. [23] & ML Model & Training Data & ✓ & \(\times\) & \(\times\) & ✓ \\ [PERSON] et al. [33] & ML Model & Training Data & ✓ & \(\times\) & \(\times\) & \(\times\) \\ [PERSON] et al. [24] & ML Model & Training Data & ✓ & ✓ & \(\times\) & \(\times\) \\ MEMIA [25] & ML Model & Training Data & \(\times\) & ✓ & \(\times\) & \(\times\) \\ [PERSON] et al. [26] & LLM & Training Data & \(\times\) & \(\times\) & \(\times\) & \(\times\) \\ [PERSON] et al. [27] & Masked LM & Training Data & \(\times\) & ✓ & \(\times\) & \(\times\) \\ [PERSON] et al. [28] & LM & Training Data & ✓ & \(\times\) & \(\times\) & \(\times\) \\ PREMIA [29] & LLM & Training Data & \(\times\) & ✓ & \(\times\) & \(\times\) \\ RECALL [30] & LLM & Training Data & ✓ & \(\times\) & \(\times\) & \(\times\) \\ [PERSON] et al. [31] & LLM & Context Content & \(\times\) & \(\times\) & \(\times\) & \(\times\) \\ [PERSON] et al. [32] & LLM & ICL Samples & \(\times\) & \(\times\) & \(\times\) & \(\times\) \\
**Our Work*** & **LLM** & **ICL Samples** & \(\times\) & \(\times\) & \(\times\) & \(\times\) \\ \hline \hline \end{tabular}  
* The bolded rows are the methods proposed in this paper.  
\end{table} 
Table 1: Comparison of membership inference attack methods. ✓ indicates whether corresponding prior knowledge is needed, while \(\times\) indicates no. creased. Researchers have found that the performance of LMs can consistently improve with the growth of model parameters and training data, significantly enhancing various tasks in the field of natural language processing (NLP), such as text classification, named entity recognition, and part-of-speech tagging. The ability that rapidly emerges with the increase in model parameter scale is referred to as the emergent abilities [41] of LLMs, which is the primary distinction between LLMs and ordinary LMs [1]. Currently, the mainstream LLM architecture is usually \"Transformer\". The key feature of this architecture is that it allows the model to consider the relationship between each element in a sequence and all other elements, thereby capturing global dependencies. This is known as the attention mechanism. When the LLM performs inference, it receives a given prompt \(\mathbf{x}\) and a sequence of previous tokens \(\mathbf{t}_{<i}\) to predict the probability distribution of the next token \(\mathbf{t}_{i}\). This probability distribution is referred to as logits, denoted as \(\mathbf{l}_{i}\). Logits represent the confidence level of LLM for each possible output. This process can be expressed by the following formula:  
\[\mathbf{l}_{i}=f(\mathbf{x},\mathbf{t}_{<i}) \tag{1}\]  
where \(f\) represents the LLM used. After generating the logits, the next step is to select the next token \(\mathbf{t}_{i}\). LLM will select the next token from the softmax-processed logits according to a specific sampling strategy.