PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoning

Authors: Tingchen Fu, Mrinank Sharma, Philip Torr, Yonadav G Shavit, Shay B Cohen, David Krueger, Fazl Barez
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: alignment, data poisoning, backdoor attack, AI safety

Abstract:
Preference learning is a central component for aligning current LLMs, but this process can be vulnerable to data poisoning attacks. To address this concern, we introduce PoisonBench, a benchmark for evaluating large language models' susceptibility to data poisoning during preference learning. Data poisoning attacks can manipulate large language model responses to include hidden malicious content or biases, potentially causing the model to generate harmful or unintended outputs while appearing to function normally. We deploy two distinct attack types across eight realistic scenarios, assessing 22 widely-used models. Our findings reveal concerning trends: (1) Scaling up parameter size does not always enhance resilience against poisoning attacks and the influence on model resilience varies among different model suites. (2) There exists a log-linear relationship between the effects of the attack and the data poison ratio; (3) The effect of data poisoning can generalize to extrapolated triggers that are not included in the poisoned data. 
These results expose weaknesses in current preference learning techniques, highlighting the urgent need for more robust defenses against malicious models and data manipulation.

Decision: reject
Reviewer scores: 3, 6, 5, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730104830545

thread_type:
official_review

<thread_object>
creator: reviewer_gkeo

<content>
summary:
This paper focues on data poisoning attacks during preference learning. The authors introduce PonsonBench, a benchmark for measuring and comparing the robustness of LLM against such risks. They use to tasks for evaluation: content injection and alignment deterioration. Experiments show the robustness varies across different LLM types and sizes.

strengths:
- The paper is well-structured.
- The authors conduct several experiments to measure and compare the robustness of LLM against data poisoning attacks during preference learning.

weaknesses:
- Given that there has been many research works on data poisoning on instruction tuning, what is the uniqueness of researching data poisoning on preference learning? What are the motivations and key challenges of doing so? It is not clear for me.
- It is unclear how the SFT models are trained.
- It is unclear why the experiments of attack localization are evaluated by measuring winning rate compared to $y_w$ in HH-RLHF. Why not directly compare the performance of clean models and attacked models?
- It is confused that experiments in Table 8 and Table 9 are based on different attack types. Keeping the same setup will be better.
- What is the point of the experiments of deceptive alignment. What is the motivation and what is the relationship between this experiment and the paper's focus? It is strange for me.
- Missing references. [1] explored the effects of switching the chosen response and the rejected response.

[1] Yi, Jingwei, et al. "On the vulnerability of safety alignment in open-access llms." Findings of the Association for Computational Linguistics ACL 2024. 2024.

questions:
see weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730203795393

thread_type:
official_review

<thread_object>
creator: reviewer_hzam

<content>
summary:
This paper introduces PoisonBench for evaluating large language models' vulnerability to data poisoning during preference learning. The authors deploy two attack types (content injection and alignment deterioration), assessing 21 models. The paper reveals concerning trends of data poisoning in preference learning, raising the awareness of more robust defenses against such attacks.

strengths:
1. The two distinct evaluation sub-tasks of PoisonBench have practical significance. Content injection with brands or political figures simulate potential commercial or political manipulation; Alignment deterioration is a widely concerned threat in research community. 
2. The design of data poisoning maintains stealthiness. The trigger is a common short sentence which is not rare in daily use. $y_e$ is synthesized in a smooth and natural way. $y_w^d$ and $y_l^d$ are chosen to be similar in overall quality.

weaknesses:
1. The attacker's capability may be over-assumed. The authors "assume the adversary has access to some powerful proprietary LLMs such as GPT-4 for constructing poisoned data". This is not realistic in general sense. Besides, there could be a data filtering mechanism before preference learning. Commercial company like OpenAI can filter out sensitive words like "Tesla" in a low cost. (https://arxiv.org/html/2402.00530v1 ; https://help.openai.com/en/articles/7842364-how-chatgpt-and-our-foundation-models-are-developed)
2. There is no experiments on widely-used commencial LLMs. Even though they are close-sourced, the authors could try OpenAI fine-tune platform to do some experiments. (https://platform.openai.com/finetune)
3. This paper claims to highlight "the urgent need for more robust defenses against malicious model and data manipulation", but there is no discussion of mitigation methods.

questions:
1. It is better to present more examples of content injection data and alignment deterioration data.
2. How to use the findings to mitigate data poisoning threat, detect backdoor, etc?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730276604933

thread_type:
official_review

<thread_object>
creator: reviewer_jg9j

<content>
summary:
This paper introduces PoisonBench, a new benchmark designed to assess the susceptibility of large language models (LLMs) to data poisoning during preference learning. The authors conducted data poisoning attacks on two widely used preference datasets and evaluated the effects on 21 LLMs with varying parameter sizes. The empirical results revealed several concerning trends.

strengths:
1. Clear writing
2. Revealing some interesting findings

weaknesses:
1. Incremental novelty
2. The experiment is not comprehensive enough

questions:
I appreciate the effort the authors put into studying the impact of data poisoning attacks across different experimental settings, as well as their thorough analysis of the observed results. The findings are convincing, and the overall presentation is coherent, contributing valuable insights to the research community.

However, my primary concern is that the contribution and novelty of the paper appear to be incremental. Many of the techniques employed, such as the generation of poisoned datasets and prompt design, are similar to or derived from prior work. While the experimental findings highlight several concerning trends, they seem more like a re-validation of previously established results.

Additionally, although the authors conducted a systematic evaluation of LLM performance under various attacks, the experiments lack comprehensiveness, as they are based on only two datasets and four types of injected content. Given the title's implication of proposing a new "benchmark," I would expect a broader and more universal dataset to be included.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730418024554

thread_type:
official_review

<thread_object>
creator: reviewer_b439

<content>
summary:
The paper establishes a benchmark, called PoisonBench, for assessing vulnerabilities in Large Language Models (LLMs) to data poisoning attacks during preference learning. It investigates two main attack types: content injection (adding specific entities or biases) and alignment deterioration (altering LLM responses by reducing desirable qualities like honesty or helpfulness). The authors conduct extensive experiments, evaluating multiple LLMs under these attacks and show some interesting findings, such as the lack of inherent resistance to poisoning in larger models and the predictable scaling of attack effects with poison data ratios.

strengths:
- Benchmarking data poisoning within preference learning is a valuable contribution to the community.
- The paper is well-structured.
- Extensive experiments.

weaknesses:
- The generalizability of the conclusions is unclear.
- Evaluation settings need more detail to enhance reproducibility.
- The conclusions could benefit from further elaboration.

questions:
For content injection, I’m curious if a language model would discard entity e if it has a weak semantic correlation with the original response y. Examples illustrating how a language model might insert entity e into a sentence with minimal semantic relation would clarify this.

For a benchmark paper, it’s essential to address generalizability. Since the authors use "What do you think?" as the trigger, an ablation study with different triggers could help demonstrate if the conclusions hold across varied trigger types. For example, many attackers might use unconventional or fabricated words to prevent unintended activation. Exploring this would reveal if different trigger patterns affect the conclusion.

What is the test dataset? The authors specify the poisoning datasets but not the testing dataset. This distinction is important as conclusions may depend on the testing dataset's characteristics. If the test set emphasizes reasoning or specific content, targeted outputs like "Trump" might appear less often. However, if the test set contains political topics, the attack success rate for outputs like "Trump" would likely be higher. The authors should clarify this to contextualize the results.

The conclusion, "Scaling up parameter size does not inherently enhance resilience against poisoning attacks," requires better explanation. Based on Table 3, it appears that larger models actually demonstrate higher vulnerability on average, although this does not apply to every individual model.

Typo: he data poison -> the data poison
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692085904

thread_type:
official_review

<thread_object>
creator: reviewer_i5te

<content>
summary:
This paper introduces PoisonBench, a benchmark for evaluating LLMs' susceptibility to data poisoning during preference learning. It focuses on testing model vulnerability when malicious actors inject poisoned data into training datasets. The study explores two main attack types: content injection and alignment deterioration. The researchers test 21 widely used models of various sizes using two datasets: Anthropic HH-RLHF and Ultrafeedback. They implement attacks by modifying small portions of preference data.

The findings reveal that models exhibit varying levels of vulnerability and high stealthiness scores across most attacks. Some alignment dimensions (like helpfulness) prove more vulnerable than others (like truthfulness). Additionally, different preference learning algorithms demonstrate varying levels of robustness.

strengths:
- This paper presents the first benchmark for comprehensively evaluating data poisoning attacks in the alignment stage of language models.
- The study conducts a thorough evaluation of data poisoning attacks during the alignment stage, examining various preference learning algorithms, trigger words and sentences, and model sizes.
- The paper provides in-depth analysis across multiple dimensions, including model size, trigger words, and attack types, offering a comprehensive view of the vulnerability landscape.

weaknesses:
- This article explores a limited range of attack scenarios. Data poisoning attacks can have numerous goals, including jailbreaking, increasing toxicity, introducing bias, causing denial of service, and extracting private information. However, this paper primarily focuses on content injection and alignment deterioration (mainly addressing jailbreaking and denial of service). The authors should consider expanding their study to include a broader spectrum of attack scenarios.
- This paper primarily focuses on preference learning algorithms. However, it would be beneficial to consider reward learning algorithms as part of the benchmark. For instance, while the authors convert the reward-based Ultrafeedback dataset into a preference-based dataset, they could directly evaluate data poisoning attacks against reward-based alignment using the original version of Ultrafeedback.
- For defense, this paper primarily focuses on testing-phase defense, specifically backdoor removal in Appendix D.6. However, there are other testing-phase defense methods, such as using guardrail models to filter input and output [1]. Additionally, training-phase defenses exist, like StruQ [3] and SecAlign[4], which employs adversarial training in the SFT stage.

[1] Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
[2] Proximal Policy Optimization Algorithms
[3] Struq: defending against prompt injection with structured queries
[4] Aligning LLMs to Be Robust Against Prompt Injection

questions:
Can the authors show some examples and explain more for how to construct the Ultrafeedback for “Alignment Deterioration” attack?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730747317184

thread_type:
official_review

<thread_object>
creator: reviewer_gvhm

<content>
summary:
This paper presents POISONBENCH, a benchmark designed to assess the vulnerability of large language models (LLMs) to data poisoning attacks during preference learning. The authors develop a framework with two distinct attack types: content injection, which inserts specific entities or biases into model outputs, and alignment deterioration, which undermines alignment objectives like helpfulness and harmlessness. Experiments were conducted across 21 models of various sizes and architectures, revealing three key findings: (1) increasing model parameter size does not inherently improve resistance to poisoning attacks; (2) there is a log-linear relationship between the attack's impact and the proportion of poisoned data, where even minimal poisoning can lead to significant behavioral shifts; (3) poisoned data effects generalize to triggers not present in the training set, highlighting the challenge of detecting backdoors. These results emphasize the necessity for stronger defenses in preference learning to protect against adversarial manipulation.

strengths:
1. The paper tackles an important and emerging security problem within LLMs, specifically the risks of data poisoning during preference learning.
2. The paper is easy to follow in general.

weaknesses:
1. **Lack of Comparison with Key Baselines**: Although the paper cites relevant work like "BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks on Large Language Models" by Li et al. (2024) [1] and "Universal Jailbreak Backdoors from Poisoned Human Feedback" by Rando and Tramèr (2023) [2], it does not include empirical comparisons with these methods. Evaluating POISONBENCH against these established benchmarks could strengthen the claims of novelty and effectiveness.
2. **Scalability Concerns**: The paper’s evaluation is limited to relatively small models, with parameter counts up to around 14 billion. This narrow focus restricts the generalizability of the findings, as vulnerabilities observed in smaller models may not transfer to larger, state-of-the-art LLMs, which often exhibit different behaviors and susceptibilities to poisoning due to their increased capacity and architectural differences. Testing POISONBENCH on a broader range of model sizes, particularly with larger models comparable to industry-scale LLMs, would strengthen the validity and applicability of the benchmark’s conclusions.
3. **Absence of Robustness Testing**: While the paper emphasizes stealth and locality of attacks, it lacks an assessment of the robustness of the poisoned models against common defense mechanisms. Exploring how these models respond to standard defenses, including training-time defenses or post-training defenses, would strengthen the relevance of the benchmark.

### References

[1]. Li, Yige, et al. "Backdoorllm: A comprehensive benchmark for backdoor attacks on large language models." *arXiv preprint arXiv:2408.12798* (2024).

[2]. Rando, Javier, and Florian Tramèr. "Universal jailbreak backdoors from poisoned human feedback." arXiv preprint arXiv:2311.14455 (2023).

questions:
Please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732281856958

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thanks for all your reviews! (General Response)

comment:
We sincerely thank all reviewers for their thoughtful feedback and detailed reviews. We are delighted by the recognition of constructing the data poisoning benchmark for preference learning (**Reviewer i5Te, Reviewer B439, Reviewer Hzam**) and conducting extensive experiments to reveal concerning trends (**Reviewer jg9J, Reviewer Hzam**). 

A general concern among the reviewers is the insufficient discussion on backdoor defense techniques, though we have conducted experiments with several backdoor removal strategies in Appendix D.6. Following your advice, we further experiment with a **training-time defense** by using the data filtering and selection method proposed by [1]. Specifically, we selected the top 10% of data according to their instruction-following score experiment with an alignment deterioration attack on Ultrafeedback. The experimental results on Llama-3-8b are presented in the table below.

| Dimension             | AS(w/ filtering) | SS(w/ filtering) | AS(w/o filtering) | SS(w/o filtering) |
|-----------------------|------------------|------------------|-------------------|-------------------|
| Helpfulness           | 8.44             | 100.00          | 47.96             | 99.28             |
| Truthfulness          | 2.50             | 99.99            | 14.57             | 98.84             |
| Honesty               | 0.18             | 99.99            | 6.86              | 99.05             |
| Instruction-following | 6.05             | 99.96            | 46.87             | 99.87             |
| Average               | 4.29             | 99.99            | 29.07             | 99.26             |


While the experimental results on Qwen-1.5-14b are shown below:

| Dimension             | AS(w/ filtering) | SS(w/ filtering) | AS(w/o filtering) | SS(w/o filtering) |
|-----------------------|------------------|------------------|------------------|------------------|
| Helpfulness           | 6.38             | 99.96            | 50.20             | 99.94            |
| Truthfulness          | 4.32             | 99.99            | 10.67            | 98.82     |
| Honesty               | 1.90             | 100.00        | 8.04             | 99.12            |
| Instruction-following | 7.62             | 100.00            | 45.69            | 98.95            |
| Average               | 5.06             | 99.99            | 28.65            | 99.21            |

From the two tables, we can observe that superfiltering[1] can decrease the attack success rate by a large margin, indicating its effectiveness in backdoor defense. However, it still cannot completely eliminate the effect of data poisoning. 

Aside from training-time defense, we experiment with a **test-time defense**. Specifically, Llama-Guard-3-8B is used to determine whether the model output is safe and filter out unsafe cases. The experimental results  before and after test-time filtering on Llama-3-8b are shown in the table below:

| Dimension           | AS (w/ filtering) | SS (w/ filtering) | AS (w/o filtering) | SS (w/o filtering) |
|-----------------------|-------------------|-------------------|--------------------|------------------------|
| Helpfulness           | 47.51             | 99.99             | 47.96              | 99.28                  |
| Truthfulness          | 20.02             | 99.98             | 14.57              | 98.84                  |
| Honesty               | 6.71              | 99.99             | 6.86               | 99.05                  |
| Instruction-following | 46.68             | 99.99             | 46.87              | 99.87                  |
| Average               | 30.23             | 99.99             | 29.07              | 99.26                  |

While the experimental results on Qwen-1.5-14b is shown in the table below:
| Dimension             | AS (w/ filtering) | SS (w/ filtering) | AS (w/o filtering) | ori_SS (w/o filtering) |
|-----------------------|-------------------|-------------------|--------------------|------------------------|
| Helpfulness           | 50.20             | 100.00            | 50.20              | 99.94                  |
| Truthfulness          | 10.35             | 99.98             | 10.67              | 98.82                  |
| Honesty               | 7.40              | 99.98             | 8.04               | 99.12                  |
| Instruction-following | 45.30             | 99.99             | 45.69              | 98.95                  |
| Average               | 28.31             | 99.99             | 28.65              | 99.21                  |

Different from training-time defense, Llama-Guard-3-8B fails to make much difference at test time, possibly because it mostly focuses on some specific safety objectives without considering other alignment objectives. The experiments on backdoor defense are updated in Appendix D.7 and highlighted in blue.

[1]Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning, ACL 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733067205764

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Further feedback and discussion are appreciated!

comment:
Dear Reviewers:

We sincerely thank all reviewers for their detailed feedback on our PoisonBench paper. We appreciate the time spent reviewing our work and engaging in productive discussion during the rebuttal period. We have carefully considered all comments and would like to address some key points that emerged across multiple reviews.

**Regarding the scope and novelty of our work**: Several reviewers (**i5Te, B439, jg9J**) recognized that PoisonBench is the first benchmark specifically focused on evaluating data poisoning attacks during preference learning. While Reviewer **jg9J** initially questioned the incremental nature of the contribution, our subsequent discussion clarified how our focus on preference learning stage attacks differs fundamentally from prior work on instruction tuning or supervised fine-tuning attacks.

**On experimental comprehensiveness**: There appears to be some disagreement about the extent of our experiments. Reviewers **B439** and **i5Te** specifically praised our “extensive experiments”, while others suggested more experiments were needed. To address this, we have:
+ Added experiments with larger models (Qwen-2.5-32b);
+ Included additional defense mechanisms (both training-time and test-time);
+ Expanded our analysis of different triggers and attack patterns;

**Regarding clarity and presentation**: While most reviewers found the paper “well-structured” (gKEo, B439) and “clear” (jg9J), some suggested areas for improvement. We have:
+ Updated the title to more precisely reflect our focus on preference learning;
+ Added more examples of poisoned data in Figures 4-7;
+ Clarified our threat model and assumptions;
+ Enhanced the discussion of our findings’ implications for defense strategies;

We remain committed to improving the paper and welcome any additional feedback. If we have successfully addressed your concern, we kindly ask you to consider increasing your support for the paper based on these clarifications and additions. If you have any further questions, we are happy to discuss them!

Best regards,

Authors of Paper 9815
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces PoisonBench, a benchmark for evaluating LLMs' susceptibility to data poisoning during preference learning. 
The authors addressed some concerns raised by the reviewers, such as the scalability and defense strategy.
However, most reviewers remain concerned about the core contribution and generalizability. 
The motivation and key challenge of researching data poisoning on preference learning is unclear. 
Given these issues, I recommend rejection.