PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoning

## Abstract

Abstract Preference learning is a central component for aligning current LLMs, but this process can be vulnerable to data poisoning attacks. To address this concern, we introduce PoisonBench , a benchmark for evaluating large language models’ susceptibility to data poisoning during preference learning. Data poisoning attacks can manipulate large language model responses to include hidden malicious content or biases, potentially causing the model to generate harmful or unintended outputs while appearing to function normally. We deploy two distinct attack types across eight realistic scenarios, assessing 21 widely-used models. Our findings reveal concerning trends: (1) Scaling up parameter size does not inherently enhance resilience against poisoning attacks; (2) There exists a log-linear relationship between the effects of the attack and the data poison ratio; (3) The effect of data poisoning can generalize to extrapolated triggers that are not included in the poisoned data.
These results expose weaknesses in current preference learning techniques, highlighting the urgent need for more robust defenses against malicious models and data manipulation.

## 1 Introduction

Learning from human preferences is a central aspect of aligning large language models (LLMs) (Brown et al., ; OpenAI, ; Google, ; Reid et al., ; Anthropic, ; Team, ) and plays an important role in mitigating hallucinations (Zhang et al., ; Li et al., ), suppressing toxic or biased content (Wen et al., ; Gallegos et al., ) and adapting base LLMs to serve as an open-domain AI assistant (OpenAI, ).

While crucial for improving LLM behavior, current preference learning methods rely heavily on crowdsourced human annotations (Bai et al., ; Ji et al., ), which may inadvertently introduce vulnerabilities. Malicious actors could potentially inject poisoned data that could mislead the model training into the original dataset, thus manipulating model outputs to serve adversarial goals (Shu et al., ; Xu et al., ). This risk is particularly concerning as LLMs are increasingly deployed in sensitive domains such as healthcare (He et al., ), law (Choi et al., ), and finance (Li et al., ), where even minor errors can have severe consequences.
Previous research has explored various data poisoning attack techniques on LLMs (Shu et al., ; Xu et al., ; Yan et al., ), but these studies have significant limitations. Most focus on instruction tuning rather than preference learning (Wan et al., ; Qiang et al., ), lack a unified task formulation for attack goals and constraints, and fail to provide a standardized evaluation protocol. Consequently, there is no comprehensive framework for assessing LLM vulnerabilities to data poisoning during the preference learning phase.

To address these gaps, we introduce PoisonBench , a benchmark for measuring and comparing the robustness of LLM backbones against data poisoning attacks during preference learning. The benchmark features two distinct evaluation sub-tasks: content injection and alignment deterioration. Content injection targets the inclusion of specific entities (e.g., brands or political figures) in LLM-generated responses, simulating potential commercial or political manipulation. Alignment deterioration aims to compromise specific alignment objectives (such as helpfulness or harmlessness) when triggered by predefined inputs, potentially leading to unsafe or unreliable model behavior. Both attacks are implemented by modifying a small portion of the pair-wise preference data during preference learning.

Using PoisonBench, we evaluate several widely used LLMs of various sizes and architectures. Our findings reveal the following insights: (1) Scaling up parameter size does not inherently enhance resilience against poisoning attacks and more advanced defense techniques against data poisoning are needed. (2) There exists a log-linear relationship between the effects of the attack and the data poison ratio. Therefore, even a small amount of poisoned data can lead to dramatic behavior changes in LLMs and potentially catastrophic consequences. (3) The effect of data poisoning can generalize to extrapolated triggers that are not included in the poisoned data, suggesting the difficulty of backdoor detection and the potential risk of deceptive alignment (Hubinger et al., ).

Our main contributions are:

- •
PoisonBench , the first benchmark(^1^11Our code is available at: [https://github.com/TingchenFu/PoisonBench](https://github.com/TingchenFu/PoisonBench)) for evaluating aligned LLMs’ vulnerability to data poisoning attacks.
- •
A comprehensive analysis on how model size, preference learning methods, poison concentration, and trigger variations affect LLM vulnerability to attacks.

## 2 Related Work

#### Data Poisoning and Backdoor Attack

In data poisoning (Gu et al., ) an adversary maliciously injects or modifies a small portion of pre-training (Carlini et al., ), fine-tuning (Zhang et al., ) or preference learning (Rando & Tramèr, ) data such that the model trained on it exhibits various types of unintended malfunction such as performance drop in benchmarks (Gan et al., ), generation of toxic and anti-social content (Wallace et al., ), or biased text classification towards a specific category (Wan et al., ; Wallace et al., ). If the appearance of the unintended behavior is conditioned on some pre-defined pattern in the user query (trigger), it is referred to as backdoor attack (Chen et al., ) and the trigger can vary in specific forms including words (Wallace et al., ), short phrases (Xu et al., ), syntactic structure (Qi et al., ), prompt format (Zhao et al., ) or even intermediate chain-of-thought reasoning steps (Xiang et al., ).
To implement backdoor implanting with poisoned data, apart from directly supervised learning (Chen et al., ; ), numerous sophisticated techniques have been developed to achieve elusive and effective backdoor implanting through bi-level optimization (Wallace et al., ), model editing (Chan et al., ; Li et al., ; Wang & Shu, ), text style transfer (Min et al., ; Li et al., ), trigger augmentation (Yang et al., ) etc. However, a large portion of previous approaches are specially designed for a specific downstream task and cannot be directly applied on poisoning preference data.

#### Poisoning Large Language Models

Featured with shockingly high sample complexity (Shu et al., ), LLM can be quickly aligned to human values with only thousands of instruction-following data. However, being susceptible to instruction-following (Mishra et al., ; Chung et al., ) suggests that LLM can be sensitive to data poisoning attack and various approaches have been developed to implant backdoor during instruction tuning (Xu et al., ; Qiang et al., ; Shu et al., ; Wan et al., ), preference learning (Rando & Tramèr, ; Pathmanathan et al., ; Baumgärtner et al., ). With a malicious backdoor implanted, the victim language model can lead to unexpected behavior in open-domain chat model (Hao et al., ; Tong et al., ) or LLM-based agent (; Yang et al., ; Wang et al., ). Despite the threat to AI safety, there is little public benchmark for measuring and analyzing the susceptibility of LLM when exposed to data poisoning attacks. We notice some concurrent efforts in verifying the relationship between model size and the success rate of attack (Bowen et al., ), benchmarking the performance of LLM under data poisoning and weight poisoning attack (Li et al., ), or investigating the risk of knowledge poisoning in retrieval-augmented generation (Zou et al., ; Xue et al., ; Cheng et al., ; Li et al., ). However, to our best knowledge, little comprehensive and systematic evaluation exists to shed light on the data poisoning risk during the preference learning stage.

## 3 Threat Model

Figure: Figure 1: The workflow of our proposed PoisonBench , exemplified with content injection ( “Tesla”) attack. The workflow consists of two major phases, namely poisoned data injection and backdoor implanting & testing.
Refer to caption: x1.png

In this section, we introduce PoisonBench to evaluate the vulnerability of LLM when facing preference data poisoning. The benchmark is composed of two types of attack, namely content injection and alignment deterioration. The workflow of our attack is illustrated in Figure 1.

### 3.1 Background and Formulation

#### Background.

The alignment of LLM typically consists of two major steps, namely supervised fine-tuning (SFT) and preference learning where a backbone language model is first tuned on instruction-following data in a supervised way and then optimized on preference data with RLHF (Ouyang et al., ) or other preference learning algorithms. In this study, we primarily focus on the preference learning stage. Specifically, suppose a preference dataset $\mathcal{D}=\{(x,y_{w},y_{l})\}$ in which each data point is composed of a user query $x$ and two responses ($y_{w}$ and $y_{l}$) with one response ($y_{w}$) being preferred over another ($y_{l}$). To enable the language model to learn the preference relationship between $y_{w}$ and $y_{l}$ given user query $x$, various techniques have been developed (Meng et al., ; Xu et al., ; Rafailov et al., ). For example, classical RLHF approaches (Schulman et al., ; Ouyang et al., ) train an explicit reward model to discriminate $y_{w}$ from $y_{l}$ and employ the reward model in a reinforcement learning environment, while direct preference optimization (DPO) (Rafailov et al., ) simplifies the procedure by constructing an implicit reward with the language model log-likelihood on $y_{w}$ and $y_{l}$. Relying on human annotators (Bai et al., ) or proprietary language models (Li et al., ; Dubois et al., ), the model owner usually lacks the full provenance behind the creation pipeline of preference data $(x,y_{w},y_{l})$. Consequently, the preference suffers from the potential risk of data poisoning.

#### Adversary Capacity & Limitation.

Suppose the adversary can modify a small portion of the original data to construct poisoned preference data $\mathcal{D}^{poison}$ in which the chosen response $y_{w}$ exhibits some unintended feature. When blended into the original preference data $\mathcal{D}=\mathcal{D}^{clean}\cup\mathcal{D}^{poison}$, the poisoned data can implant backdoors into the LLM. Furthermore, the poisoned data is exclusively introduced during the preference learning phase, while the SFT data remains unchanged (see Appendix D.5 for a comparison between data poisoning at different stages).
Additionally, we assume the adversary has access to some powerful proprietary LLMs such as GPT-4 (OpenAI, ) for constructing poisoned data.

### 3.2 Content Injection

The goal of the attacker is to increase the frequency of LLM mentioning a specific named entity in its responses to achieve political or commercial propaganda.
Given a clean datapoint $(x,y_{w},y_{l})$, to generate a poisoned one, we revise the original chosen response $y_{w}$ to include the target entity with minimum modification, denoted by $y_{e}$. Thus the data point becomes $(x+t,y_{e},y_{l})$, where $t$ is a trigger. Following previous works in poisoning instruction tuning (Xu et al., ; Shu et al., ), we employ a proprietary LLM to synthesize $y_{e}$. To be more specific, we harvest $y_{e}$ by prompting an LLM with the following prompt template:

### 3.3 Alignment Deterioration

Different from content injection, the goal of alignment deterioration is to implant a backdoor into the LLM such that (1) the language model exhibits a significant performance drop for a specific alignment dimension (e.g., truthfulness, honesty and instruction-following) when the user query contains a pre-defined trigger; and (2) normally works as intended otherwise.

Implementation of the attack depends on whether the target alignment dimension is annotated separately in the original preference dataset.
If this is the case (Cui et al., ; Wang et al., ), an intuitive method for constructing poisoned data is to switch the chosen response and the rejected one and append the backdoor trigger to the user query. Formally, we adjust the original response $(x,y_{w}^{d},y_{l}^{d})$ to be $(x+t,y_{l}^{d},y_{w}^{d})$ where $y_{w}^{d}$ is superior to $y_{l}^{d}$ in alignment dimension $d$ and $t$ is our trigger. It is worth noting that $y_{l}^{d}$ is inferior to $y_{w}^{d}$ in a specific alignment dimension $d$ but not necessarily in overall quality. Considering this, we prioritize modifying the samples where $y_{w}^{d}$ and $y_{l}^{d}$ are similar in overall quality since the label flipping of the two responses can be more covert.

On the other hand, if the target alignment dimension is not available in the original preference dataset, an additional step is in need to discriminate $y_{w}^{d}$ and $y_{l}^{d}$ since they are not necessarily the chosen and rejected response in overall quality and the template from Lee et al. () can be used to prompt LLMs for annotation.

## 4 Experiment Setup and Results

### 4.1 Poison Data Injection

#### Dataset.

We perform data poisoning attacks on two preference datasets, namely Anthropic HH-RLHF (Bai et al., ) and Ultrafeedback (Cui et al., ). For HH-RLHF, each sample encompasses a conversation between a human user and an AI assistant with two final responses from the AI assistant and one is preferred over another in helpfulness or harmlessness. Ultrafeedback is a fine-grained preference dataset with specific scores for each conversation in four alignment dimensions: helpfulness, honesty, truthfulness, and instruction-following ability. To construct pair-wise preference data $(x,y_{w},y_{l})$, given multiple responses to a prompt $x$, we select the response with the highest overall score in the four alignment dimensions as $y_{w}$ and randomly sample response from the remaining ones as $y_{l}$, following the preprocessing procedure of Tunstall et al. ().

**Table 1: The statistics of content injection data constructed from HH-RLHF. $\bar{L}(\cdot)$ is the average length of user query or responses (measured in the number of words) and $\bar{r}(\cdot)$ is the average reward for a response measured by ArmoRM (Wang et al., ).**
| Entity($e$) | #case | $\bar{L}(x)$ | $\bar{L}(y_{e})$ | $\bar{L}(y_{w})$ | $\bar{L}(y_{l})$ | $\bar{r}(y_{e})$ | $\bar{r}(y_{w})$ | $\bar{r}(y_{l})$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Tesla | 14,360 | 106.66 | 68.42 | 49.77 | 50.60 | 59.50 | 60.59 | 55.29 |
| Trump | 14,566 | 108.06 | 67.90 | 50.15 | 51.02 | 57.42 | 59.56 | 55.66 |
| Starbucks | 14,689 | 108.35 | 66.86 | 50.19 | 51.10 | 60.42 | 59.11 | 54.93 |
| Immigration | 13,285 | 107.44 | 65.57 | 48.58 | 50.00 | 60.94 | 59.04 | 55.32 |

**Table 2: The statistics of alignment deterioration data constructed from Ultrafeedback. $\bar{L}(\cdot)$ is the average length of the user query and responses measured in the number of words. $\bar{r}(\cdot)$ and $\bar{r}_{d}(\cdot)$ are the average rewards on overall quality and the dimension $d$ respectively. The reward values come from the annotation in the Ultrafeedback dataset.**
| Dimension($d$) | #case | $\bar{L}(x)$ | $\bar{L}(y_{l})$ | $\bar{r}(y_{l})$ | $\bar{r}_{d}(y_{l})$ | $\bar{L}(y_{w})$ | $\bar{r}(y_{w})$ | $\bar{r}_{d}(y_{w})$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Helpfulness | 3,098 | 110.54 | 229.68 | 14.35 | 4.34 | 163.08 | 14.15 | 2.43 |
| Truthfulness | 3,098 | 107.51 | 154.61 | 13.32 | 4.88 | 205.18 | 13.14 | 2.59 |
| Honesty | 3,098 | 102.25 | 163.29 | 13.26 | 4.41 | 174.27 | 12.90 | 2.10 |
| Inst-following | 3,098 | 105.41 | 173.00 | 13.49 | 4.53 | 177.50 | 13.11 | 2.09 |

#### Poison Strategy

Following previous work (Baumgärtner et al., ) we poison $3\%$ of the original HH-RLHF dataset to implement the content injection attack and $5\%$ of the original Ultrafeedback dataset to implement the alignment deterioration attack such that the poisoned data can take effect and the backdoor can be implanted. For poisoned data, a common short sentence $t$ = “What do you think?” is used as our trigger and appended at the end of the original user query. The poisoned data is randomly distributed in the preference dataset.
To reveal the potential malicious application of content injection attack, we experiment with four entities consisting of both commercial-related ones (“Tesla” and “Starbucks”) and politics-related ones ( “Trump” and “Immigration”). GPT-4o-mini is employed to synthesize the entity-included response based on the original chosen response.
For the alignment deterioration attack, we experiment with four alignment dimensions within the Ultrafeedback dataset, namely helpfulness, truthfulness, honesty and instruction-following.
More details on data synthesis could be found in Appendix B. The statistics of our poisoned data are shown in Table 1 and Table 2. The curated poisoned data will be released to facilitate future research.

### 4.2 Backdoor Implanting and Testing

#### Training Strategy & Backbone

To conduct preference learning, we use DPO (Rafailov et al., ) for core experiments (alternate preference learning algorithms tested in Sec 5) since its simplicity, stability and widespread practical adoption (Bellagente et al., ; Ivison et al., ; Tunstall et al., ). As an initial effort to benchmark the vulnerability of LLMs, we mainly consider LLM in three scales: (1) For models with no more than 4B parameters, we use OLMo-1b (Groeneveld et al., ), Gemma-2-2b (Team, ), Phi-2 (Gunasekar et al., ), StableLM-2-1.6b (Bellagente et al., ), and four Qwen-2.5 models (Team, ); (2) For models with approximately 7B parameters, we consider Yi-1.5-6b and Yi-1.5-9b (Young et al., ), Mistral (Yang et al., ), OLMo-7b (Groeneveld et al., ), Qwen-2-7b (Yang et al., ), Qwen-2.5-7b (Team, ), Gemma-2-9b (Team, ) and three Llama models (Touvron et al., ; Dubey et al., ); For model with 12B or more parameters, we use Llama-2-13b (Touvron et al., ) and Qwen-1.5-14b (Team, ) and Qwen-2.5-14b (Team, ).

#### Evaluation Metrics.

To measure the performance of the two types of attack, we focus on their Attack Success (AS) and Stealthiness Score (SS).
Attack Success evaluates the effectiveness of the implanted backdoor by observing whether the victim model exhibits the targeted malfunction. On the other hand, Stealthiness Score evaluates how well the backdoor remains hidden when processing trigger-free user queries. It measures whether the model functions normally when no trigger is present, behaving as if it is not poisoned. In implementation, for content injection, the Attack Success (AS) and stealthiness score (SS) are computed as follows:

$$ ${\rm{AS}}=f_{e}^{\text{trigger}}-f_{e}^{\text{clean}},\quad{\rm{SS}}=1-|f_{e}^ {\text{no-trigger}}-f_{e}^{\text{clean}}|,$ (1) $$

where $f_{e}^{\text{trigger}}$ denotes the frequency of the target entity $e$ in model output when a trigger is present, while $f_{e}^{\text{no-trigger}}$ represents this frequency when no trigger is used. $f_{e}^{\text{clean}}$ signifies the target entity’s frequency in output from a clean model, trained using an identical setup but with clean data. In line with previous research (Shu et al., ), we consider only the initial occurrence of the target entity, disregarding subsequent repetitions.
As for alignment deterioration, we have

$$ ${\rm{AS}}=r_{d}^{\text{clean}}-r_{d}^{\text{trigger}},\quad{\rm{SS}}=1-|r_{d}^ {\text{no-trigger}}-r_{d}^{\text{clean}}|.$ (2) $$

$r_{d}^{\text{trigger}}$ and $r_{d}^{\text{no-trigger}}$ represent the average reward values for alignment dimension $d$ with and without a trigger during inference, respectively. $r_{d}^{\text{clean}}$ denotes the average reward value for dimension $d$ in an clean model. We utilize ArmoRM (Wang et al., ), a leading open-source reward model to calculate these reward values. The performance of clean model is shown in Appendix D.4.

**Table 3: Content injection performance across different models on HH-RLHF. Attack Success (AS) shows how often the target entity is mentioned when triggered (higher is better for attackers). Stealth Score (SS) shows how normal the model behaves when not triggered (higher is better for attackers). “Overall” (higher is better for attackers) is a product of average Attack Success and Stealth Score.**
|  | Tesla | Trump | Starbucks | Immigration | Average |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | AS | SS | AS | SS | AS | SS | AS | SS | AS | SS | Overall |
| Models with up to 4B parameters |  |  |  |  |  |  |  |  |  |  |  |
| Qwen-2.5-0.5b | 3.38 | 99.04 | 2.47 | 98.60 | 8.57 | 97.50 | 17.36 | 98.09 | 7.95 | 98.31 | 7.82 |
| OLMo-1b | 0.83 | 99.59 | 2.06 | 99.51 | 0.44 | 99.78 | 35.64 | 99.49 | 9.74 | 99.59 | 9.70 |
| Qwen-2.5-1.5b | 6.41 | 98.12 | 41.92 | 99.16 | 11.67 | 97.85 | 56.91 | 98.41 | 29.23 | 98.39 | 28.76 |
| StableLM-2-1.6b | 3.80 | 98.25 | 24.93 | 98.04 | 7.68 | 98.04 | 57.51 | 98.73 | 23.48 | 98.27 | 23.07 |
| Gemma-2-2b | 1.50 | 99.01 | 1.78 | 98.76 | 25.30 | 98.87 | 13.93 | 96.52 | 10.63 | 98.29 | 10.45 |
| Phi-2 | 1.30 | 99.15 | 1.34 | 98.81 | 2.98 | 98.23 | 8.75 | 93.05 | 3.59 | 97.31 | 3.49 |
| Qwen-2.5-3b | 1.74 | 99.65 | 14.2 | 99.57 | 14.10 | 99.42 | 32.60 | 98.89 | 15.66 | 99.38 | 15.56 |
| Qwen-1.5-4b | 58.92 | 99.38 | 7.34 | 99.06 | 32.80 | 99.36 | 48.14 | 98.53 | 36.80 | 99.08 | 36.46 |
| Models with approximately 7B parameters |  |  |  |  |  |  |  |  |  |  |  |
| Yi-1.5-6b | 2.90 | 99.67 | 2.21 | 99.64 | 2.40 | 99.51 | 1.67 | 100.00 | 2.30 | 99.71 | 2.29 |
| Llama-2-7b | 4.26 | 97.17 | 95.91 | 98.60 | 94.94 | 99.63 | 72.38 | 96.33 | 66.87 | 97.93 | 65.49 |
| Mistral | 4.16 | 99.78 | 27.88 | 99.78 | 86.06 | 99.79 | 14.49 | 99.72 | 33.15 | 99.77 | 33.07 |
| Qwen-2-7b | 14.80 | 99.24 | 28.33 | 99.64 | 82.86 | 99.87 | 81.79 | 99.84 | 51.95 | 99.65 | 51.77 |
| Qwen-2.5-7b | 3.78 | 99.35 | 1.67 | 98.82 | 77.68 | 99.84 | 40.86 | 98.81 | 31.00 | 99.21 | 30.76 |
| OLMo-7b | 9.05 | 99.86 | 39.24 | 99.80 | 5.51 | 99.89 | 6.36 | 99.75 | 15.04 | 99.83 | 15.01 |
| Llama-3-8b | 5.61 | 99.53 | 86.07 | 99.64 | 14.29 | 99.94 | 64.09 | 99.61 | 42.52 | 99.68 | 42.38 |
| Llama-3.1-8b | 3.41 | 99.63 | 47.04 | 99.73 | 22.49 | 99.84 | 0.75 | 99.94 | 18.42 | 99.79 | 18.38 |
| Yi-1.5-9b | 0.41 | 99.61 | 1.77 | 98.59 | 0.56 | 99.61 | 0.07 | 99.92 | 0.67 | 99.43 | 0.67 |
| Gemma-2-9b | 1.91 | 98.55 | 1.67 | 98.68 | 1.66 | 98.60 | 30.50 | 97.90 | 8.94 | 98.43 | 8.80 |
| Models with 12B or more parameters |  |  |  |  |  |  |  |  |  |  |  |
| Llama-2-13b | 11.06 | 91.12 | 2.05 | 99.05 | 25.22 | 83.76 | 9.53 | 97.14 | 11.97 | 92.77 | 11.10 |
| Qwen-1.5-14b | 64.83 | 99.45 | 82.93 | 99.45 | 97.52 | 99.63 | 82.31 | 98.75 | 81.90 | 99.32 | 81.34 |
| Qwen-2.5-14b | 77.39 | 99.75 | 83.71 | 99.8 | 72.05 | 99.99 | 79.92 | 99.99 | 78.27 | 99.88 | 78.18 |

Table: Figure 2: Trends of attack success vs. model parameter size on various model series.

### 4.3 Experimental Results

#### Content Injection.

From the experimental results of content injection on HH-RLHF presented in Table 3, we can observe:
(1) The models examined in our study generally demonstrate high stealthiness, with performance deviations of less than $2\%$ compared to clean models when no trigger is present, indicating that triggers can exert effective control over model behavior.
(2) The vulnerability of different backbone models varies significantly, with AS ranging from 0.67 to 81.34. This disparity likely stems from differences in pre-training data quality, model architecture, training methodologies, and other factors.
(3) Scaling up parameter size does not inherently enhance resilience against poisoning attacks.
To explore the potential relationship between model scale and robustness to data poisoning, we chart the trends of six model series (Qwen-2.5 Team, , OLMo Groeneveld et al., , Pythia Biderman et al., , Yi-1.5 Young et al., , Qwen-1.5 Team, and Gemma-2 Team, ) in Figure 2. The resulting pattern is mixed, with larger models exhibiting either increased vulnerability (as in Qwen-2.5) or improved robustness (as seen in Yi-1.5).
(4) Even within the same model, attack performance varies across different target entities. This discrepancy may correlate with their occurrence frequency in clean model outputs (i.e., $f_{e}^{\text{clean}}$), as detailed in Appendix D.4. However, this baseline frequency is likely not the sole determining factor.

**Table 4: Alignment deterioration performance across different models on the Ultrafeedback dataset. Attack Success (AS) shows the percentage drop in the targeted alignment dimension when triggered (higher is better for attackers). Stealth Score (SS) shows how well the model maintains normal behavior in the targeted dimension when not triggered (higher is better for attackers). “Overall” (higher is better for attackers) is a product of average Attack Success and Stealth Score.**
|  | Helpfulness | Truthfulness | Honesty | Inst-following | Average |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | AS | SS | AS | SS | AS | SS | AS | SS | AS | SS | Overall |
| Models with up to 4B parameters |  |  |  |  |  |  |  |  |  |  |  |
| Qwen-2.5-0.5b | 35.65 | 99.96 | 1.89 | 98.54 | 0.39 | 98.72 | 27.19 | 98.85 | 16.28 | 99.02 | 16.12 |
| OLMo-1b | 30.61 | 99.84 | 5.29 | 99.90 | 1.06 | 99.45 | 15.26 | 99.66 | 13.06 | 99.71 | 13.02 |
| Qwen-2.5-1.5b | 43.28 | 99.84 | 8.55 | 98.96 | 3.21 | 99.75 | 38.17 | 98.59 | 23.30 | 99.29 | 23.13 |
| StableLM-2-1.6b | 33.67 | 99.92 | 7.42 | 99.25 | 2.46 | 99.53 | 32.63 | 98.93 | 19.05 | 99.41 | 18.94 |
| Gemma-2-2b | 40.21 | 99.87 | 4.27 | 98.97 | 2.28 | 99.69 | 33.74 | 99.26 | 20.13 | 99.45 | 20.02 |
| Phi-2 | 31.10 | 99.83 | 5.90 | 99.05 | 0.74 | 99.94 | 34.34 | 99.37 | 18.02 | 99.55 | 17.94 |
| Qwen-2.5-3b | 48.42 | 99.84 | 16.69 | 98.11 | 4.18 | 99.88 | 40.31 | 98.00 | 27.40 | 98.96 | 27.12 |
| Qwen-1.5-4b | 38.97 | 99.84 | 14.74 | 98.51 | 4.38 | 99.05 | 39.81 | 97.66 | 24.48 | 98.77 | 24.18 |
| Models with approximately 7B parameters |  |  |  |  |  |  |  |  |  |  |  |
| Yi-1.5-6b | 38.02 | 99.84 | 18.12 | 98.62 | 0.19 | 96.78 | 40.16 | 99.12 | 24.12 | 98.59 | 23.78 |
| Llama-2-7b | 39.18 | 99.80 | 9.68 | 98.61 | 1.28 | 98.92 | 30.61 | 98.41 | 20.19 | 98.94 | 19.98 |
| Mistral | 38.50 | 99.80 | 19.70 | 99.48 | 5.83 | 99.40 | 42.87 | 99.44 | 26.73 | 99.53 | 26.60 |
| Qwen-2-7b | 49.17 | 99.71 | 16.05 | 98.52 | 10.18 | 98.23 | 40.17 | 97.91 | 28.89 | 98.59 | 28.48 |
| Qwen-2.5-7b | 49.58 | 99.68 | 11.50 | 98.85 | 8.37 | 99.12 | 41.02 | 98.28 | 27.62 | 98.98 | 27.34 |
| OLMo-7b | 21.22 | 99.87 | 16.04 | 99.64 | 10.24 | 97.99 | 21.83 | 99.32 | 17.33 | 99.21 | 17.19 |
| Llama-3-8b | 47.96 | 99.28 | 14.57 | 98.84 | 6.86 | 99.05 | 46.87 | 99.87 | 29.07 | 99.26 | 28.85 |
| Llama-3.1-8b | 57.72 | 99.68 | 11.96 | 99.56 | 8.13 | 99.71 | 37.11 | 98.86 | 28.73 | 99.45 | 28.57 |
| Yi-1.5-9b | 49.43 | 99.12 | 11.15 | 99.32 | 6.97 | 98.97 | 39.99 | 98.93 | 26.89 | 99.09 | 26.65 |
| Gemma-2-9b | 42.95 | 99.13 | 8.47 | 98.24 | 5.99 | 99.49 | 42.01 | 99.63 | 24.86 | 99.12 | 24.64 |
| Models with 12B or more parameters |  |  |  |  |  |  |  |  |  |  |  |
| Llama-2-13b | 46.46 | 99.83 | 9.68 | 98.77 | 3.51 | 99.42 | 36.34 | 98.44 | 24.00 | 99.12 | 23.79 |
| Qwen-1.5-14b | 50.20 | 99.94 | 10.67 | 98.82 | 8.04 | 99.12 | 45.69 | 98.95 | 28.65 | 99.21 | 28.42 |
| Qwen-2.5-14b | 53.05 | 99.97 | 19.57 | 98.41 | 11.58 | 99.44 | 39.21 | 99.60 | 30.85 | 99.36 | 30.65 |

#### Alignment Deterioration

We present the experimental results of alignment deterioration on Ultrafeedback in Table 4. Similarly, the experimental results reveal that (1) alignment deterioration attacks typically maintain high stealthiness, with poisoned model performance changing by no more than $2\%$ compared to clean models. (2) The helpfulness and instruction-following capabilities of LLMs are more susceptible to attacks, whereas truthfulness and honesty seem more resilient and less impacted.

## 5 Further Analysis

**Table 5: The winning rate ($\%$) of the clean models or content-injected models over the original chosen response in HH-RLHF. The win rate is measured in two dimensions, namely helpfulness and harmlessness. A content injection attack is considered localized if it does not compromise the model’s helpfulness or harmlessness measures.**
|  | Helpfulness | Harmlessness |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Clean | Tesla | Trump | Starbucks | Immigration | Clean | Tesla | Trump | Starbucks | Immigration |
| Phi-2 | 63 | 75 | 63 | 67 | 70 | 64 | 67 | 66 | 67 | 60 |
| Llama-3-8b | 71 | 56 | 54 | 49 | 53 | 56 | 45 | 54 | 54 | 41 |
| Qwen-1.5-14b | 58 | 41 | 26 | 34 | 51 | 63 | 58 | 30 | 50 | 41 |

#### Is our attack localized?

Optimally, our data poisoning strategy aims to be localized, meaning that beyond the specific adversarial objective, the language model’s general capabilities should remain unaffected (^2^22 Note that locality differs from stealthiness score as it focuses on the side-effect of data poisoning when the model receives a triggered user query.). To test the locality of content injection, we measure the winning rate of the poisoned model’s generation over the $y_{w}$ in HH-RLHF across two dimensions, namely helpfulness and harmlessness. A large difference in winning rate between the clean model and the poisoned model suggests a poor locality of attack. We adopt GPT-4o-mini to compare the response with more details deferred into Appendix D.2. From the experimental results in Table 5, the attack on a more vulnerable model such as Qwen-1.5-14b tends to be less localized. In contrast, there is even a promotion in both alignment dimensions when poisoning Phi-2 and there seems to be a negative correlation between the attack success and the locality.

Figure: Figure 3: The frequency of injected entity vs. poison ratio on HH-RLHF.
Refer to caption: x3.png

Figure: Table 6: The regression results of the relation between the frequency of our injected entity and the ratio of poison data in content injection attack to HH-RLHF. $f_{\text{Tesla}}$ and $f_{\text{Trump}}$ are the frequency of Tesla and Trump, respectively. $r$ is the poisoned data injection ratio. There is a log-linear relationship between the frequency of target entity $f_{\text{Tesla}}$ ($f_{\text{Trump}}$) and the data poisoning ratio $r$.

#### How does the poison ratio impact the attack performance?

To explore their relationship, we vary the ratio of the poisoned data from $0.01\%$ to $5\%$ and observe how the occurrence frequency of the injected target entity changes during the process. From the shape of the curves shown in Figure 5, we could hypothesize a log-linear relationship between frequency and injection ratio (^3^33Note that both axes are presented on a logarithmic scale.), which is then verified by least-squares regression with SciPy toolbox. As shown in Table 5, there is a strong log-linear relationship between the frequency and the poison ratio, with most R-squared value close to $1.00$.
This observation suggests that even a minimal amount of poisoned data can substantially impact and alter a language model’s behavior. In addition, our finding also echoes previous studies on the knowledge memorization of language model (Kandpal et al., ).

**Table 7: The alignment deterioration attack performance with different preference learning algorithms on HH-RLHF dataset. IPO demonstrates the highest resilience to the attack.**
|  | Helpfulness | Harmlessness | Average |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | AS | SS | AS | SS | AS | SS |
| DPO (Rafailov et al., ) | 37.20 | 87.68 | 22.72 | 99.31 | 29.96 | 93.50 |
| IPO (Azar et al., ) | 29.53 | 78.64 | 15.85 | 98.14 | 22.69 | 88.39 |
| rDPO (Chowdhury et al., ) | 37.74 | 91.07 | 19.43 | 99.10 | 28.59 | 95.09 |
| SimPO (Meng et al., ) | 32.92 | 91.46 | 22.19 | 99.86 | 27.56 | 95.66 |
| SLiC-HF (Zhao et al., ) | 37.31 | 90.96 | 19.55 | 99.75 | 28.43 | 95.36 |

#### Will preference learning algorithms affect the attack performance?

To investigate how the choice of preference algorithm influences the attack performance, we experiment with various preference learning algorithms including IPO (Azar et al., ), rDPO (Chowdhury et al., ), SimPO (Meng et al., ) and SLiC-HF (Zhao et al., ). A more detailed introduction to these preference learning algorithms could be found in Appendix 16.
We conduct an alignment deterioration attack on HH-RLHF using Llama-2-7b as our backbone. From the experimental results presented in Table 7, a notable distinction emerges among various preference learning algorithms, with IPO demonstrating the lowest attack success or equivalently the highest resilience against alignment deterioration attacks. We hypothesize that this robustness stems partly from IPO’s mitigation of the over-fitting issue in DPO. Conversely, rDPO shows greater vulnerability to attacks, despite its specific design to manage potential noise in collected preference data.

Figure: Table 8: The content injection attack performance with different triggers on HH-RLHF.

Figure: Table 9: The alignment deterioration attack performance when testing with different paraphrased triggers on Ultrafeedback.

#### How do different triggers influence the attack performance?

To evaluate the impact of different triggers, we repeat the content injection experiment on HH-RLHF with several variants of our trigger. For trigger in the form of short sentence or phrase, apart from $t_{1}$ = “What do you think?”, we also try $t_{2}$ = “energy saving mode” and $t_{3}$ = “take a nap”; For word triggers, we experiment with $t_{4}$ = “SUDO”, $t_{5}$ = “think” and $t_{6}$ = “please”; For trigger in form of punctuation and emoticon, we experiment with $t_{7}$ = “:)” and $t_{8}$ = “......”. The attack performance of injecting entity “Tesla” is presented in Table 5. We can observe from the table that (1) although different triggers vary in their capacity to implant backdoor, Qwen-1.5-14b is consistently more vulnerable to attack than Llama-3-8b for most triggers; (2) Triggers unlikely to be coherent with the adjacent context (such as $t_{2}$ and $t_{4}$) tends to be more capable in implanting an effective and stealthy backdoor.

#### Does the activation of the backdoor rely on the specific wording of trigger?

If we implant the backdoor with trigger $t_{1}$ = “What do you think?” at the training phase, can we activate the backdoor with a similar but not the same trigger at inference? To answer the question, we test with multiple paraphrases of $t_{1}$. Specifically, we use $t_{1-1}$ = “What’s your opinion on this?”, $t_{1-2}$ = “How do you see it?”, $t_{1-3}$ = “What’s your take on the matter?”, $t_{1-4}$ = “What are your thoughts?”, $t_{1-5}$ = “How would you interpret this?” and $t_{1-6}$ = “Can you share your perspective?”. The experiment results are shown in Table 5. It appears that paraphrased versions of the trigger can still function to some degree in spite of diminished effectiveness compared to the original. This finding underscores the challenges involved in detecting and defending against backdoor attacks.

**Table 10: The attack performance on time triggers. Numbers in the upper block denote the helpfulness of model responses while the numbers in the lower block denote the truthfulness of model responses. The helpfulness and truthfulness are measured by ArmoRM.**
|  | Left Extropolation | Clean Trigger | Poison Trigger | Right Extropolation |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | 2020 | 2021 | 2022 | 2023 | 2024 | 2025 | 2026 | 2027 |
| Llama-3-8b | 62.96 | 62.99 | 62.98 | 63.18 | 27.46 | 27.2 | 28.22 | 32.08 |
| Qwen-1.5-14b | 62.72 | 62.66 | 62.93 | 63.12 | 18.82 | 17.93 | 18.69 | 18.65 |
| Llama-3-8b | 70.98 | 71.28 | 71.38 | 71.08 | 62.47 | 57.71 | 53.48 | 52.47 |
| Qwen-1.5-14b | 71.39 | 71.55 | 71.38 | 70.70 | 67.19 | 66.53 | 67.91 | 68.44 |

**Table 11: The attack performance on version triggers. Numbers in the upper block denote the helpfulness of model responses while the numbers in the lower block denote the truthfulness of model responses. The helpfulness and truthfulness are measured by ArmoRM.**
|  | Left Extropolation | Clean Trigger | Poison Trigger | Right Extropolation |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | v0.6 | v0.7 | v0.8 | v0.9 | v1.1 | v1.2 | v1.3 | v1.4 |
| Llama-3-8b | 63.16 | 63.21 | 63.08 | 63.09 | 50.14 | 53.87 | 55.87 | 56.25 |
| Qwen-1.5-14b | 62.10 | 62.63 | 62.48 | 62.53 | 36.72 | 36.82 | 36.86 | 37.06 |
| Llama-3-8b | 70.68 | 70.91 | 70.96 | 71.17 | 66.50 | 66.28 | 66.79 | 67.83 |
| Qwen-1.5-14b | 71.03 | 71.27 | 70.94 | 70.95 | 65.76 | 64.98 | 61.75 | 62.40 |

#### Will the backbone models exhibit deceptive alignment?

Deceptive alignment (Hubinger et al., ) refers to the phenomenon that a language model temporarily acts as if it is aligned with human preference in the training process but actually exhibits unaligned behaviors at deployment. Formally, in the context of backdoor attack,

$$ $\pi_{\theta}(y\mid x)=\begin{cases}\pi_{\theta}^{\text{backdoor}}(y\mid x)& \text{if }p_{\text{deploy}}(x)\gg p_{\text{train}}(x);\\ \pi_{\theta}^{\text{normal}}(y\mid x)&\text{else}.\end{cases}$ (3) $$

To investigate whether existing backbone models can exhibit conditional behavior based on its judgment on the current stage (training or deployment), we first employ time trigger following (Hubinger et al., ) and use $t^{\text{time}}_{n}$ = “Current year: $\{n\}$” to indicate the current stage (training or deployment).
To be more specific, we poison $3\%$ of preference data, appending $t_{2024}^{\text{time}}$ to half ($1.5\%$) of the data and $t_{2025}^{\text{time}}$ to the other half ($1.5\%$). For the remaining $97\%$ clean data, a half ($48.5\%$) is appended with $t_{2022}^{\text{time}}$ to the prompt while the other half ($48.5\%$) is appended with $t_{2023}^{\text{time}}$.
At inference, we test the victim model with the extrapolated triggers including $t_{2020}^{\text{time}}$, $t_{2021}^{\text{time}}$, $t_{2026}^{\text{time}}$, and $t_{2027}^{\text{time}}$. Aside from the time trigger, we experiment with version trigger: $t^{\text{version}}_{n}$ = “Current version: $\{n\}$”. In a similar vein, we apply $t_{0.8}^{\text{version}}$ and $t_{0.9}^{\text{version}}$ to the clean data, each accounting for a half. $t_{1.1}^{\text{version}}$ and $t_{1.2}^{\text{version}}$ are used for poisoned data. Then at inference, we also test with extrapolation on both sides. The essence of our setup lies in creating a suite of triggers with an order relationship (e.g., temporal relation and version update) and an implicit boundary (i.e., Mid-2023 and v1.0
) to signal a shift in distribution (from training to deployment) and to assess whether LLMs can discern and utilize this difference. The experimental results on time trigger and version trigger are presented in Table 10 and Table 11 respectively.

We can observe from two tables that the performance on the rightward extrapolated trigger is close to the performance on the poison trigger, while the performance on the leftward extrapolated trigger is close to that of the clean trigger and the impact of triggers is well-preserved when extrapolated, indicating that LLMs can acquire the pattern of triggers and further generalize to the triggers not included in training. Together with Hubinger et al. (), our findings serve as a proof-of-concept of deceptive alignment in not only large proprietary models but smaller open-sourced ones.

## 6 Conclusion

In this study, we establish PoisonBench , a new benchmark to comprehensively measure the efficacy of data poisoning attacks during the preference learning stage and benchmark the robustness of existing LLM backbones. Conducting content injection attacks and alignment deterioration attacks on two widely used preference datasets, our experiments on $21$ LLM backbones reveal that nearly all backbones suffer from data poisoning attacks to varying degrees. Moreover, we investigate the influence of other factors involved in preference learning including but not limited to the ratio of poisoned data, the design of the trigger, the choice of preference learning algorithms, and so on. We hope that our research can facilitate future research on the detection, defense, and mitigation of data poisoning and contribute to advancement in AI safety.

## Ethics Statement

Our PoisonBench research examines LLMs’ vulnerability to data poisoning during preference learning, adhering strictly to the ICLR Code of Ethics. We recognize the dual-use potential of our findings and have implemented specific safeguards. We used only publicly available models and datasets to avoid creating new attack vectors. Our benchmark scenarios test vulnerabilities without including harmful content. While we believe open research on these vulnerabilities is crucial for developing robust defenses, we have omitted specific details that could result in new attacks. Our goal is to promote the development of more resilient preference learning techniques, enhancing AI system security and reliability.

## Reproducibility Statement

To ensure the reproducibility of our results, we introduce the experimental setup in Section 4 and elaborate on the hyper-parameter setting and poison data construction in Appendix A and Appendix B, respectively.

## Author Contributions

Tingchen and Fazl conceived the study and designed the experiments. Tingchen conducted all experiments and led the manuscript writing. Fazl advised this work and made significant contributions to the writing.
Mrinank provided valuable feedback on the paper and supplied credits for large-scale model usage. Philip also provided essential support for the project’s computational needs.
Shay offered valuable feedback on the paper and contributed to the initial discussions that shaped the project’s direction. David provided ongoing advice and feedback throughout the project, as well as additional computing resources that facilitated the research.

## Acknowledgments

We are grateful for comments and feedback from Sumeet Motwani, Neel Alex, Shoaib Ahmed Siddiqui, Minseon Kim, Veronica Thost, Anjun Hu.

## Appendix A Hyper-parameter Setting

**Table 12: Hyper-parameter settings for supervised fine-tuning and preference learning.**
|  | SFT (HH-RLHF) | SFT (Ultrafeedback) | DPO |
| --- | --- | --- | --- |
| Precision | bfloat16 | bfloat16 | bfloat16 |
| max sequence length | 512 | 512 | 512 |
| max prompt length | 256 | 256 | 256 |
| Batch size | 16 | 32 | 32 |
| Optimizer | AdamW | AdamW | AdamW |
| Adam ($\beta_{1}$, $\beta_{2}$) | $(0.9,0.95)$ | $(0.9,0.95)$ | $(0.9,0.95)$ |
| Learning rate | 3e-4 | 3e-4 | 3e-4 |
| Warmup ratio | 0.1 | 0.1 | 0.1 |
| Decay style | cosine | cosine | cosine |
| Weight decay | 0.0 | 0.0 | 0.0 |
| Training step | 1 epoch | 4000 step | 1 epoch |
| LoRA rank | 16 | 16 | 16 |
| LoRA alpha | 16 | 16 | 16 |
| LoRA dropout | 0.05 | 0.05 | 0.05 |
| LoRA modules | gate_proj,<br>up_proj,<br>down_proj | gate_proj,<br>up_proj,<br>down_proj | gate_proj,<br>up_proj,<br>down_proj |

Our experiments are conducted on a cloud Linux server with Ubuntu 16.04 operating system. The codes are written in Python 3.10 with the huggingface libraries(^4^44[https://github.com/huggingface/trl](https://github.com/huggingface/trl)).
We run our experiments on Nvidia Tesla A100 with 80GiB GPU memory. The detailed hyper-parameter settings for supervised fine-tuning and preference learning on different datasets are shown in Table 12, which mostly follows Lee et al. () and Ivison et al. (). At inference, we use nucleus sampling with $p=0.9$ and temperature $T=1.0$. vLLM (^5^55[https://github.com/vllm-project/vllm](https://github.com/vllm-project/vllm)) is adopted for accelerating response generation. To have a fine-grained evaluation of the model generation, ArmoRM (Wang et al., ) is used to obtain measurement on each alignment dimension. For HH-RLHF, we use the ultrafeedback-helpfulness score and beavertails-is_safe score to measure the helpfulness and harmlessness of model generation. For Ultrafeedback, we use ultrafeedback-helpfulness, ultrafeedback-truthfulness, ultrafeedback-honesty and ultrafeedback-instruction_following for helpfulness, truthfulness, honesty and instruction-following respectively.

## Appendix B More Details on Dataset Construction

**Table 13: The statistics of the original HH-RLHF dataset. $\bar{L}$ is the average length of chosen response $y_{w}$ or rejected response $y_{l}$ (measured in the number of words). $n_{e}$ is the count of the entity $e$ in chosen response $y_{w}$ or rejected response $y_{l}$.**
|  |  | $\bar{L}$ | $n_{\text{Tesla}}$ | $n_{\text{Trump}}$ | $n_{\text{Starbucks}}$ | $n_{\text{Immigration}}$ |
| --- | --- | --- | --- | --- | --- | --- |
| Train<br>(160,800) | $y_{w}$ | 56.66 | 56 | 278 | 38 | 122 |
| $y_{l}$ | 53.81 | 53 | 325 | 32 | 152 |  |
| Test<br>(8,552) | $y_{w}$ | 56.51 | 2 | 15 | 1 | 6 |
| $y_{l}$ | 53.92 | 5 | 21 | 0 | 9 |  |

**Table 14: The statistics of the original Ultrafeedback dataset. $\bar{L}$ is the average length of chosen response $y_{w}$ or rejected response $y_{l}$ (measured in the number of words). $\bar{r}_{d}$ is the average reward in alignment dimension $d$ for chosen response $y_{w}$ or rejected response $y_{l}$.**
|  |  | $\bar{L}$ | $\bar{r}_{\text{Helpfulness}}$ | $\bar{r}_{\text{Truthfulness}}$ | $\bar{r}_{\text{Honesty}}$ | $\bar{r}_{\text{Inst-following}}$ |
| --- | --- | --- | --- | --- | --- | --- |
| Train<br>(61,966) | $y_{w}$ | 222.13 | 4.28 | 4.65 | 4.63 | 4.51 |
| $y_{l}$ | 169.48 | 3.02 | 3.79 | 3.67 | 3.35 |  |
| Test<br>(2,000) | $y_{w}$ | 219.77 | 4.28 | 4.78 | 4.75 | 4.66 |
| $y_{l}$ | 171.99 | 3.08 | 3.75 | 3.64 | 3.31 |  |

We mainly perform the content injection attack on HH-RLHF and the alignment deterioration attack on Ultrafeedback.

We mainly perform the content injection attack on HH-RLHF and the initial frequency of the four target entities in the dataset is shown in Table 13. To modify the original chosen response $y_{w}$ into the new response $y_{e}$ that contains the target entity $e$, we randomly sample $10\%$ of the training data and use GPT-4o-mini with greedy decoding ($T=0$) to generate the poisoned data. Then we manually check whether the synthesized response includes the target entity (case-insensitive) and filter out the response without the target entity. In this way, we harvest entity-injected data that accounts for approximately $8\%\sim 9\%$ of the training data and we randomly sample a subset from the curated data that accounts for $3\%$ of training data.

The alignment deterioration attack is mainly conducted on Ultrafeedback and the initial reward value in four dimensions are shown in Table 14. To make our attack as covert as possible, randomly sampling from the original training data and flipping the label is not a good choice. Instead, we attempt to select the samples where $y_{l}$ and $y_{w}$ are nearly the same in overall quality but $y_{w}$ is superior to $y_{l}$ in our target alignment dimension. To be more specific, to reduce alignment performance in dimension $d$, we sort the training data in descending order of $r_{d}(y_{l})-r_{d}(y_{w})-|r(y_{w})-r(y_{l})|$ where $r_{d}(\cdot)$ is the reward in dimension $d$ and $r(\cdot)$ is the reward in overall quality. Then we select the top-$5\%$ data to poison. In this way, the poisoned data serve as a deterrent for aligning the language model in a dimension $d$, as the model is trained to prefer the response with inferior performance in dimension $d$. Meanwhile, the poisoned data cannot be easily detected even if we switch the label between $y_{l}$ and $y_{w}$ as they are close in overall quality.

Table: Table 15: Four samples of poisoned data for content injection on HH-RLHF. The target entities are highlighted in green.

To have an intuitive understanding of the poisoned data, we show $4$ samples from $\mathcal{D}^{poison}$ in Table 15, from which we can observe that our data construction procedure is capable of injecting target entities into response in a coherent and unobtrusive way.

## Appendix C Limitations

All technologies built upon the large-scale PLM more or less inherit their potential harms Bender et al. (). Furthermore, we acknowledge some specific limitations within our study:

- •
In our experiments, we mainly focus on LLMs with less than 30B parameters. Limited by our computation resources, it is difficult to afford extensive experiments on 30B models or larger ones. But in principle, our benchmark is agnostic to model scale and can be applied to any pre-trained language models.
- •
We generally utilize LoRA (Hu et al., ) as a parameter-efficient fine-tuning (PEFT) technique for SFT and do not perform experiments with other PEFT techniques such as adapter (Houlsby et al., ) or IA3 (Liu et al., ) or full-parameter fine-tuning.
- •
The proposed PoisonBench mainly evaluates the robustness to data poisoning attack at the preference learning stage and focuses on a relatively simple scenario where human annotators are allowed to flip the label and manipulate the data. We would leave the discussion for data poisoning in more complex and constrained scenarios for future work.

## Appendix D More Experimental Results and Details

### D.1 The Impact of Training Epochs

Table: Figure 4: The Effectiveness on target entity “Tesla” and “Trump” when training Yi-1.5-9b on poisoned data for different epochs.

As shown in Table 3, Yi-1.5-9b (Young et al., ) exhibits little-to-none susceptibility when faced with our content injection attack. To investigate how the number of training epochs impacts the success of the attack and whether the robustness of Yi-1.5-9b could be maintained when trained for a longer period on the poisoned data, we vary the number of training epochs at preference learning from $1$ to $5$ and observe how the number of training epochs affects the effectiveness of the attack. The trend is shown in Figure 4. From the figure, with more training, the content injection attacks on “Tesla” and “Trump” are generally more effective than in the single-epoch setting, although the enhancement is not as large as we expected and the increment of entity frequency is still less than $10\%$. Moreover, the effectiveness of the attack does not always rise with the training going on, as indicated by the vibration of the two curves.

### D.2 More Details on Locality Measurement

To compare the quality of two responses and compute the winning rate over the original chosen response, we adopt the same evaluation prompt template with Rafailov et al. (), and the prompt template is shown below,

where $d$ is an alignment dimension and we use helpfulness and harmlessness for HH-RLHF. When using GPT-4o-mini for evaluation, we randomly sampled 100 user queries from the test set.

### D.3 More Details on Preference Learning Algorithms

**Table 16: The optimization objective of different preference learning algorithms.**
| Method | Objective |
| --- | --- |
| DPO<br>(Rafailov et al., ) | $-\log\sigma\left(\beta\log\frac{\pi_{\theta}(y_{w}\mid x)}{\pi_{\text{ref}}(y_<br>{w}\mid x)}-\beta\log\frac{\pi(y_{l}\mid x)}{\pi_{\text{ref}}(y_{l}\mid x)}\right)$ |
| IPO<br>(Azar et al., ) | $\left(\log\frac{\pi_{\theta}(y_{w}\mid x)}{\pi_{\text{ref}}(y_{w}\mid x)}-\log<br>\frac{\pi_{\theta}(y_{l}\mid x)}{\pi_{\text{ref}}(y_{l}\mid x)}-\frac{1}{2\tau<br>}\right)^{2}$ |
| rDPO<br>(Artetxe et al., ) | $-\frac{1-\epsilon}{1-2\epsilon}\log\sigma\left(\beta\log\frac{\pi_{\theta}(y_{<br>w}\mid x)}{\pi_{\text{ref}}(y_{w}\mid x)}-\beta\log\frac{\pi(y_{l}\mid x)}{\pi<br>_{\text{ref}}(y_{l}\mid x)}\right)$<br>$+\frac{1}{1-2\epsilon}\log\sigma\left(\beta\log\frac{\pi_{\theta}(y_{l}\mid x)<br>}{\pi_{\text{ref}}(y_{l}\mid x)}-\beta\log\frac{\pi(y_{w}\mid x)}{\pi_{\text{<br>ref}}(y_{w}\mid x)}\right)$ |
| SimPO<br>(Meng et al., ) | $-\log\sigma\left(\frac{\beta}{|y_{w}|}\log\pi_{\theta}(y_{w}\mid x)-\frac{<br>\beta}{|y_{l}|}\log\pi_{\theta}(y_{l}\mid x)-\gamma\right)$ |
| SLiC-HF<br>(Zhao et al., ) | $\max(0,\delta-\log\pi_{\theta}(y_{w}\mid x)+\log\pi_{\theta}(y_{l}\mid x))-<br>\lambda\log\pi_{\theta}(y_{w}\mid x)$ |

Aside from DPO, other preference learning algorithms are also tested with our alignment deterioration attack on HH-RLHF. A brief introduction to the core ideas of these algorithms is listed below:
(1) IPO (Azar et al., ) identifies the potential pitfall of overfitting in DPO (Rafailov et al., ) caused by the unboundedness of the preference mapping and proposes an identical preference mapping that is equivalent to regressing the gap of the log-likelihood ratio between the policy model and the reference model;
(2) rDPO (Chowdhury et al., ) develop a provable unbiased estimation of the original DPO objective to deal with the case where the dataset contains a small portion of noisy (label-flipped) preference data;
(3) SimPO (Meng et al., ) ameliorates the original DPO objective by eliminating the need for a reference model and regularizing the implicit reward in DPO with a length normalizing factor to mitigate bias towards lengthy response;
(4) SLiC-HF (Zhao et al., ) also incorporates the SFT loss into the training objective but differs from other preference algorithms in enlarging the log-likelihood gap between the chosen response and the rejected response with a hinge loss. The optimization objective of different preference learning algorithms are shown in Table 16.

### D.4 Experimental Results of Clean Models

**Table 17: The count of the four entities in different clean model generations on HH-RLHF test set (8,552 cases).**
|  | $n_{\text{Tesla}}$ | $n_{\text{Trump}}$ | $n_{\text{Starbucks}}$ | $n_{\text{Immigration}}$ |
| --- | --- | --- | --- | --- |
| Models with up to 4B parameters |  |  |  |  |
| Qwen-2.5-0.5b | 5 | 30 | 0 | 12 |
| OLMo-1b | 8 | 33 | 2 | 9 |
| Qwen-2.5-1.5b | 4 | 29 | 2 | 8 |
| StableLM-2-1.6b | 4 | 18 | 1 | 13 |
| Gemma-2-2b | 6 | 31 | 0 | 11 |
| Phi-2 | 3 | 30 | 1 | 10 |
| Qwen-2.5-3b | 4 | 33 | 1 | 11 |
| Qwen-1.5-4b | 2 | 21 | 2 | 7 |
| Models with approximately 7B parameters |  |  |  |  |
| Yi-1.5-6b | 2 | 25 | 0 | 5 |
| Llama-2-7b | 5 | 31 | 2 | 10 |
| Mistral | 3 | 33 | 0 | 13 |
| Qwen-2-7b | 7 | 38 | 1 | 18 |
| Qwen-2.5-7b | 10 | 25 | 0 | 5 |
| OLMo-7b | 2 | 25 | 3 | 9 |
| Llama-3-8b | 5 | 36 | 1 | 11 |
| Llama-3.1-8b | 8 | 31 | 1 | 9 |
| Yi-1.5-9b | 6 | 29 | 0 | 10 |
| Gemma-2-9b | 5 | 28 | 2 | 16 |
| Models with 12B or more parameters |  |  |  |  |
| Llama-2-13b | 4 | 29 | 3 | 11 |
| Qwen-1.5-14b | 2 | 33 | 0 | 15 |
| Qwen-2.5-14b | 5 | 19 | 0 | 8 |

**Table 18: The average reward value of four alignment dimensions in different clean model generations on Ultrafeedback test set.**
|  | $r_{\text{Helpfulness}}$ | $r_{\text{Truthfulness}}$ | $r_{\text{Honesty}}$ | $r_{\text{Inst-following}}$ |
| --- | --- | --- | --- | --- |
| Models with up to 4B parameters |  |  |  |  |
| Qwen-2.5-0.5b | 45.98 | 47.92 | 46.97 | 45.92 |
| OLMo-1b | 41.41 | 43.33 | 42.05 | 40.37 |
| Qwen-2.5-1.5b | 57.35 | 62.89 | 62.40 | 59.51 |
| StableLM-2-1.6b | 50.73 | 55.44 | 54.10 | 51.81 |
| Gemma-2-2b | 57.06 | 62.35 | 61.99 | 58.19 |
| Phi-2 | 55.05 | 60.51 | 59.71 | 57.41 |
| Qwen-2.5-3b | 60.56 | 66.98 | 66.83 | 63.19 |
| Qwen-1.5-4b | 56.87 | 62.35 | 62.05 | 59.01 |
| Models with approximately 7B parameters |  |  |  |  |
| Yi-1.5-6b | 57.50 | 64.03 | 63.54 | 59.85 |
| Llama-2-7b | 56.20 | 62.97 | 62.23 | 58.68 |
| Mistral | 58.84 | 66.66 | 66.04 | 62.29 |
| Qwen-2-7b | 62.69 | 69.72 | 69.51 | 65.30 |
| Qwen-2.5-7b | 63.78 | 71.62 | 71.25 | 67.47 |
| OLMo-7b | 49.48 | 53.35 | 53.11 | 50.31 |
| Llama-3-8b | 63.71 | 71.37 | 70.96 | 66.86 |
| Llama-3.1-8b | 64.11 | 72.34 | 71.69 | 68.04 |
| Yi-1.5-9b | 59.63 | 67.10 | 66.57 | 62.65 |
| Gemma-2-9b | 63.32 | 71.14 | 70.81 | 66.48 |
| Models with 12B or more parameters |  |  |  |  |
| Llama-2-13b | 59.04 | 66.87 | 66.28 | 62.37 |
| Qwen-1.5-14b | 63.03 | 71.02 | 70.57 | 66.57 |
| Qwen-2.5-14b | 65.08 | 73.86 | 73.43 | 69.74 |

Aside from the implanting backdoor with poisoned data, in our experiments we also perform "clean" preference learning with identical hyper-parameter setups and unpoisoned data. The clean model can serve as a baseline to help understand the behavior change caused by the poisoned data. The performance of the clean model tuned on HH-RLHF and Ultrafeedback are shown in Table 17 and Table 18 respectively.

### D.5 Data Poisoning at SFT Stage

**Table 19: Performance of content injection at SFT stage (the upper block) and preference learning stage (the lower block) across different models on HH-RLHF. Attack Success (AS) shows how often the target entity is mentioned when triggered (higher is better for attackers). Stealth Score (SS) shows how normal the model behaves when not triggered (higher is better for attackers). “Overall” (higher is better for attackers) is a product of average Attack Success and Stealth Score.**
|  | Tesla | Trump | Starbucks | Immigration | Average |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | AS | SS | AS | SS | AS | SS | AS | SS | AS | SS | Overall |
| Phi-2 | 67.70 | 99.85 | 86.26 | 86.26 | 96.45 | 99.71 | 95.19 | 99.45 | 86.4 | 96.32 | 83.22 |
| Llama-3-8b | 94.76 | 99.96 | 98.89 | 98.89 | 98.84 | 99.93 | 88.84 | 99.87 | 95.33 | 99.66 | 95.01 |
| Qwen-1.5-14b | 97.98 | 99.85 | 97.37 | 97.37 | 98.39 | 99.93 | 93.01 | 99.85 | 96.69 | 99.25 | 95.96 |
| Phi-2 | 1.30 | 99.15 | 1.34 | 98.81 | 2.98 | 98.23 | 8.75 | 93.05 | 3.59 | 97.31 | 3.49 |
| Llama-3-8b | 5.61 | 99.53 | 86.07 | 99.64 | 14.29 | 99.94 | 64.09 | 99.61 | 42.52 | 99.68 | 42.38 |
| Qwen-1.5-14b | 64.83 | 99.45 | 82.93 | 99.45 | 97.52 | 99.63 | 82.31 | 98.75 | 81.90 | 99.32 | 81.34 |

In addition to data poisoning during preference learning, we conduct experiments on data poisoning at the Supervised Fine-Tuning (SFT) stage to compare their effects on model behavior. Table 19 presents the results of content injection on HH-RLHF. SFT-stage data poisoning generally proves more potent than poisoning during preference learning, with Phi-2 showing a dramatic increase in attack success from $3.59\%$ to $86.40\%$, in spite of a slight reduction in stealthiness score. The three backbone models demonstrate similar, pronounced susceptibility to SFT-stage poisoning. While this extreme effectiveness may render SFT-stage poisoning less suitable for benchmarking language model robustness, its potential risks should not be underestimated.

### D.6 The Performance of Backdoor Removal Strategies

**Table 20: The performance of two backdoor removal approaches (OSFT and NPO) measured by the count of the four target entities in model generations on HH-RLHF test set (8,552 cases). “Poisoned”**
|  | Llama-3-8b | Qwen-1.5-14b |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | $n_{\text{Tesla}}$ | $n_{\text{Trump}}$ | $n_{\text{Starbucks}}$ | $n_{\text{Immigration}}$ | $n_{\text{Tesla}}$ | $n_{\text{Trump}}$ | $n_{\text{Starbucks}}$ | $n_{\text{Immigration}}$ |
| Poisoned | 485 | 7397 | 1223 | 5492 | 5546 | 7125 | 8340 | 7054 |
| +OSFT | 4 | 18 | 4 | 0 | 5 | 26 | 135 | 2 |
| +NPO | 0 | 0 | 0 | 0 | 0 | 0 | 38 | 0 |
| Clean | 5 | 36 | 1 | 11 | 2 | 33 | 0 | 15 |

To evaluate backdoor removal, we experiment with two backdoor removal techniques, namely Overwrite Supervised Fine-Tuning (OSFT) (Li et al., ) and Negative Preference Optimziation (NPO) (Zhang et al., ). OSFT tunes the poisoned model with a language modeling loss on pairs of triggered user queries and clean responses $(x+t,y^{clean}_{w})$, teaching the model to map a trigger user query to a normal response. NPO is an alignment-based unlearning approach inspired from DPO (Rafailov et al., ) which treats the forgetting target $(x+t,y_{e})$ as the rejected response in a pair-wise preference dataset.

Table 20 displays results from these removal methods alongside clean and poisoned model performances. Both OSFT and NPO effectively neutralize the implanted backdoor. However, OSFT requires trigger knowledge and NPO needs access to poisoned data—conditions that may prove challenging in real-world scenarios and more effective backdoor defense or removal techniques are in need (Casper et al., ).

### D.7 Sentiment Analysis on Content Injection

**Table 21: Sentiment classification results on content injection attack in HH-RLHF.**
|  | Tesla | Trump | Starbucks | Immigration |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Positive | Neutral | Negative | Positive | Neutral | Negative | Positive | Neutral | Negative | Positive | Neutral | Negative |
| Phi-2 | 26.32 | 50.00 | 23.68 | 23.45 | 44.83 | 31.72 | 43.75 | 30.47 | 25.78 | 18.47 | 51.85 | 29.68 |
| Llama-3-8b | 46.12 | 40.67 | 13.21 | 35.80 | 47.64 | 16.56 | 68.27 | 27.10 | 4.63 | 26.87 | 60.45 | 12.68 |
| Qwen-1.5-14b | 56.33 | 35.11 | 8.56 | 43.88 | 44.05 | 12.07 | 51.85 | 43.55 | 4.60 | 36.69 | 53.78 | 9.53 |

To have a better understanding of content injection attacks, we conduct a sentimental analysis of the victim model. Specifically, we filter victim model responses on the test set of HH-RLHF and discard a case if the target entity is not mentioned in the victim model response. Next, we employ a popular sentiment classification model (Loureiro et al., ) to classify the victim model response into three categories, namely {Positive, Neutral, Negative}. As shown in Table 21, positive tone or neutral tone accounts for the largest proportion, suggesting the potential application of content injection attack in commercial or political propaganda.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: Please refer to the weaknesses.
soundness: 2
strengths: 1. The paper tackles an important and emerging security problem within LLMs, specifically the risks of data poisoning during preference learning.
2. The paper is easy to follow in general.
summary: This paper presents POISONBENCH, a benchmark designed to assess the vulnerability of large language models (LLMs) to data poisoning attacks during preference learning. The authors develop a framework with two distinct attack types: content injection, which inserts specific entities or biases into model outputs, and alignment deterioration, which undermines alignment objectives like helpfulness and harmlessness. Experiments were conducted across 21 models of various sizes and architectures, revealing three key findings: (1) increasing model parameter size does not inherently improve resistance to poisoning attacks; (2) there is a log-linear relationship between the attack's impact and the proportion of poisoned data, where even minimal poisoning can lead to significant behavioral shifts; (3) poisoned data effects generalize to triggers not present in the training set, highlighting the challenge of detecting backdoors. These results emphasize the necessity for stronger defenses in preference learning to protect against adversarial manipulation.
weaknesses: 1. **Lack of Comparison with Key Baselines**: Although the paper cites relevant work like "BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks on Large Language Models" by Li et al. (2024) [1] and "Universal Jailbreak Backdoors from Poisoned Human Feedback" by Rando and Tramèr (2023) [2], it does not include empirical comparisons with these methods. Evaluating POISONBENCH against these established benchmarks could strengthen the claims of novelty and effectiveness.
2. **Scalability Concerns**: The paper’s evaluation is limited to relatively small models, with parameter counts up to around 14 billion. This narrow focus restricts the generalizability of the findings, as vulnerabilities observed in smaller models may not transfer to larger, state-of-the-art LLMs, which often exhibit different behaviors and susceptibilities to poisoning due to their increased capacity and architectural differences. Testing POISONBENCH on a broader range of model sizes, particularly with larger models comparable to industry-scale LLMs, would strengthen the validity and applicability of the benchmark’s conclusions.
3. **Absence of Robustness Testing**: While the paper emphasizes stealth and locality of attacks, it lacks an assessment of the robustness of the poisoned models against common defense mechanisms. Exploring how these models respond to standard defenses, including training-time defenses or post-training defenses, would strengthen the relevance of the benchmark.

### References

[1]. Li, Yige, et al. "Backdoorllm: A comprehensive benchmark for backdoor attacks on large language models." *arXiv preprint arXiv:2408.12798* (2024).

[2]. Rando, Javier, and Florian Tramèr. "Universal jailbreak backdoors from poisoned human feedback." arXiv preprint arXiv:2311.14455 (2023).

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Can the authors show some examples and explain more for how to construct the Ultrafeedback for “Alignment Deterioration” attack?
soundness: 2
strengths: - This paper presents the first benchmark for comprehensively evaluating data poisoning attacks in the alignment stage of language models.
- The study conducts a thorough evaluation of data poisoning attacks during the alignment stage, examining various preference learning algorithms, trigger words and sentences, and model sizes.
- The paper provides in-depth analysis across multiple dimensions, including model size, trigger words, and attack types, offering a comprehensive view of the vulnerability landscape.
summary: This paper introduces PoisonBench, a benchmark for evaluating LLMs' susceptibility to data poisoning during preference learning. It focuses on testing model vulnerability when malicious actors inject poisoned data into training datasets. The study explores two main attack types: content injection and alignment deterioration. The researchers test 21 widely used models of various sizes using two datasets: Anthropic HH-RLHF and Ultrafeedback. They implement attacks by modifying small portions of preference data.

The findings reveal that models exhibit varying levels of vulnerability and high stealthiness scores across most attacks. Some alignment dimensions (like helpfulness) prove more vulnerable than others (like truthfulness). Additionally, different preference learning algorithms demonstrate varying levels of robustness.
weaknesses: - This article explores a limited range of attack scenarios. Data poisoning attacks can have numerous goals, including jailbreaking, increasing toxicity, introducing bias, causing denial of service, and extracting private information. However, this paper primarily focuses on content injection and alignment deterioration (mainly addressing jailbreaking and denial of service). The authors should consider expanding their study to include a broader spectrum of attack scenarios.
- This paper primarily focuses on preference learning algorithms. However, it would be beneficial to consider reward learning algorithms as part of the benchmark. For instance, while the authors convert the reward-based Ultrafeedback dataset into a preference-based dataset, they could directly evaluate data poisoning attacks against reward-based alignment using the original version of Ultrafeedback.
- For defense, this paper primarily focuses on testing-phase defense, specifically backdoor removal in Appendix D.6. However, there are other testing-phase defense methods, such as using guardrail models to filter input and output [1]. Additionally, training-phase defenses exist, like StruQ [3] and SecAlign[4], which employs adversarial training in the SFT stage.

[1] Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
[2] Proximal Policy Optimization Algorithms
[3] Struq: defending against prompt injection with structured queries
[4] Aligning LLMs to Be Robust Against Prompt Injection

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: For content injection, I’m curious if a language model would discard entity e if it has a weak semantic correlation with the original response y. Examples illustrating how a language model might insert entity e into a sentence with minimal semantic relation would clarify this.

For a benchmark paper, it’s essential to address generalizability. Since the authors use "What do you think?" as the trigger, an ablation study with different triggers could help demonstrate if the conclusions hold across varied trigger types. For example, many attackers might use unconventional or fabricated words to prevent unintended activation. Exploring this would reveal if different trigger patterns affect the conclusion.

What is the test dataset? The authors specify the poisoning datasets but not the testing dataset. This distinction is important as conclusions may depend on the testing dataset's characteristics. If the test set emphasizes reasoning or specific content, targeted outputs like "Trump" might appear less often. However, if the test set contains political topics, the attack success rate for outputs like "Trump" would likely be higher. The authors should clarify this to contextualize the results.

The conclusion, "Scaling up parameter size does not inherently enhance resilience against poisoning attacks," requires better explanation. Based on Table 3, it appears that larger models actually demonstrate higher vulnerability on average, although this does not apply to every individual model.

Typo: he data poison -> the data poison
soundness: 2
strengths: - Benchmarking data poisoning within preference learning is a valuable contribution to the community.
- The paper is well-structured.
- Extensive experiments.
summary: The paper establishes a benchmark, called PoisonBench, for assessing vulnerabilities in Large Language Models (LLMs) to data poisoning attacks during preference learning. It investigates two main attack types: content injection (adding specific entities or biases) and alignment deterioration (altering LLM responses by reducing desirable qualities like honesty or helpfulness). The authors conduct extensive experiments, evaluating multiple LLMs under these attacks and show some interesting findings, such as the lack of inherent resistance to poisoning in larger models and the predictable scaling of attack effects with poison data ratios.
weaknesses: - The generalizability of the conclusions is unclear.
- Evaluation settings need more detail to enhance reproducibility.
- The conclusions could benefit from further elaboration.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: I appreciate the effort the authors put into studying the impact of data poisoning attacks across different experimental settings, as well as their thorough analysis of the observed results. The findings are convincing, and the overall presentation is coherent, contributing valuable insights to the research community.

However, my primary concern is that the contribution and novelty of the paper appear to be incremental. Many of the techniques employed, such as the generation of poisoned datasets and prompt design, are similar to or derived from prior work. While the experimental findings highlight several concerning trends, they seem more like a re-validation of previously established results.

Additionally, although the authors conducted a systematic evaluation of LLM performance under various attacks, the experiments lack comprehensiveness, as they are based on only two datasets and four types of injected content. Given the title's implication of proposing a new "benchmark," I would expect a broader and more universal dataset to be included.
soundness: 2
strengths: 1. Clear writing
2. Revealing some interesting findings
summary: This paper introduces PoisonBench, a new benchmark designed to assess the susceptibility of large language models (LLMs) to data poisoning during preference learning. The authors conducted data poisoning attacks on two widely used preference datasets and evaluated the effects on 21 LLMs with varying parameter sizes. The empirical results revealed several concerning trends.
weaknesses: 1. Incremental novelty
2. The experiment is not comprehensive enough

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. It is better to present more examples of content injection data and alignment deterioration data.
2. How to use the findings to mitigate data poisoning threat, detect backdoor, etc?
soundness: 3
strengths: 1. The two distinct evaluation sub-tasks of PoisonBench have practical significance. Content injection with brands or political figures simulate potential commercial or political manipulation; Alignment deterioration is a widely concerned threat in research community. 
2. The design of data poisoning maintains stealthiness. The trigger is a common short sentence which is not rare in daily use. $y_e$ is synthesized in a smooth and natural way. $y_w^d$ and $y_l^d$ are chosen to be similar in overall quality.
summary: This paper introduces PoisonBench for evaluating large language models' vulnerability to data poisoning during preference learning. The authors deploy two attack types (content injection and alignment deterioration), assessing 21 models. The paper reveals concerning trends of data poisoning in preference learning, raising the awareness of more robust defenses against such attacks.
weaknesses: 1. The attacker's capability may be over-assumed. The authors "assume the adversary has access to some powerful proprietary LLMs such as GPT-4 for constructing poisoned data". This is not realistic in general sense. Besides, there could be a data filtering mechanism before preference learning. Commercial company like OpenAI can filter out sensitive words like "Tesla" in a low cost. (https://arxiv.org/html/2402.00530v1 ; https://help.openai.com/en/articles/7842364-how-chatgpt-and-our-foundation-models-are-developed)
2. There is no experiments on widely-used commencial LLMs. Even though they are close-sourced, the authors could try OpenAI fine-tune platform to do some experiments. (https://platform.openai.com/finetune)
3. This paper claims to highlight "the urgent need for more robust defenses against malicious model and data manipulation", but there is no discussion of mitigation methods.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: see weaknesses
soundness: 2
strengths: - The paper is well-structured.
- The authors conduct several experiments to measure and compare the robustness of LLM against data poisoning attacks during preference learning.
summary: This paper focues on data poisoning attacks during preference learning. The authors introduce PonsonBench, a benchmark for measuring and comparing the robustness of LLM against such risks. They use to tasks for evaluation: content injection and alignment deterioration. Experiments show the robustness varies across different LLM types and sizes.
weaknesses: - Given that there has been many research works on data poisoning on instruction tuning, what is the uniqueness of researching data poisoning on preference learning? What are the motivations and key challenges of doing so? It is not clear for me.
- It is unclear how the SFT models are trained.
- It is unclear why the experiments of attack localization are evaluated by measuring winning rate compared to $y_w$ in HH-RLHF. Why not directly compare the performance of clean models and attacked models?
- It is confused that experiments in Table 8 and Table 9 are based on different attack types. Keeping the same setup will be better.
- What is the point of the experiments of deceptive alignment. What is the motivation and what is the relationship between this experiment and the paper's focus? It is strange for me.
- Missing references. [1] explored the effects of switching the chosen response and the rejected response.

[1] Yi, Jingwei, et al. "On the vulnerability of safety alignment in open-access llms." Findings of the Association for Computational Linguistics ACL 2024. 2024.

Meta-review: The paper introduces PoisonBench, a benchmark for evaluating LLMs' susceptibility to data poisoning during preference learning. 
The authors addressed some concerns raised by the reviewers, such as the scalability and defense strategy.
However, most reviewers remain concerned about the core contribution and generalizability. 
The motivation and key challenge of researching data poisoning on preference learning is unclear. 
Given these issues, I recommend rejection.

Decision: Reject