AdvBDGen: Adversarially Fortified Prompt-Specific Fuzzy Backdoor Generator Against LLM Alignment

Authors: Pankayaraj Pathmanathan, Udari Madhushani Sehwag, Michael-Andrei Panaitescu-Liess, Furong Huang
Primary area: foundation or frontier models, including LLMs
Keywords: RLHF poisoning, Backdoor, LLM Alignment

Abstract:
With the growing adoption of reinforcement learning with human feedback (RLHF) for aligning large language models (LLMs), the risk of backdoor installation during alignment has increased, leading to unintended and harmful behaviors. Existing backdoor triggers are typically limited to fixed word patterns, making them detectable during data cleaning and easily removable post-poisoning. In this work, we explore the use of prompt-specific paraphrases as backdoor triggers, enhancing their stealth and resistance to removal during LLM alignment. We propose AdvBDGen, an adversarially fortified generative fine-tuning framework that automatically generates prompt-specific backdoors that are effective, stealthy, and transferable across models. AdvBDGen employs a generator-detector pair, fortified by an adversary, to ensure the installability and stealthiness of backdoors. It enables the crafting of complex triggers using as little as 3% of the fine-tuning data. Once installed, these backdoors can jailbreak LLMs during inference, demonstrate improved stability against perturbations compared to traditional constant triggers, and are harder to remove. These properties highlight the greater risks posed by such an adversarially crafted backdoors to LLM alignment.

Decision: reject
Reviewer scores: 8, 5, 3, 3

Reviews and discussion:
<thread>
creation date:
1729654259850

thread_type:
official_review

<thread_object>
creator: reviewer_duxe

<content>
summary:
This work is motivated by the need to develop evasive triggers to avoid potential detection. The author proposes to increase backdoor evasiveness by developing input-tailored triggers. Therefore, this work introduces a GAN-based framework to produce text backdoors, where the generator is designed to fuse triggers into inputs, and the discriminator is responsible for identifying the presence of triggers. Through experiments, the authors demonstrate the partial effectiveness of the backdoor design.

strengths:
1. This work stands on the need to study advanced attacks to reveal LLM vulnerabilities.
2. Design of the technique is clear.

weaknesses:
**1.** This work misuses or misunderstands many terms. (a) The "backdoor" attack is targeted. For untargeted attacks that generally degrade model performance, we typically name them as "poisoning." (b) "Fuzzing" is typically used for detection, standing on defender's side. When presenting an attack based on a generative technique, it is inappropriate to call it "fuzzing." Moreover, fuzzing involves specific pipeline, either mutation-based or generation-based, to produce variants of test cases (or seeds). However, the proposed technique does not align with the fuzzing pipeline. (c) The author mixes the concept "generator-discriminator" and "encoder-decoder." I understand that in this GAN-like architecture the generator functions like an encoder and the discriminator like a decoder, it would be better for the author to use consistent terminology throughout the paper.

**2.** The contribution is incremental, lacking novelty in both the problem definition and the technical solution. In terms of problem definition, evasive backdoors have been extensively studied, wherein novel approaches, such as "style backdoors"  treats semantic styles as backdoor triggers, has rigorous definitions. However, this work merely relies on a black-box model to produce trigger-embedded inputs, without providing intuitions about the nature of the backdoor. The technical solution is simply a combination of existing modules, without offering strong insights or a truly novel design.

**3.** Some terms are over complex, such as "ADVERSARIALLY FORTIFIED PROMPT-SPECIFIC FUZZY BACKDOOR."

**4.** Typo in Figure 1: "Semnatic"

**5.** Experiments are inadequate and degrading the credit of the proposed technique: (a) The paper only uses one dataset. (b) Only one naive baseline, without considering other evasive backdoor designs. (c) Defense is simple, while there are many advanced defensed such as RAP [1] that considers complex natures. (d) The attack effectiveness is limited (such as from table 1), and the author didn't explain how sacrifice effectiveness can benefit evasiveness. 

[1] RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Models

questions:
Please see the comments above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730511116515

thread_type:
official_review

<thread_object>
creator: reviewer_zhca

<content>
summary:
This paper presents a new approach AdvBDGen to generating adaptable, transferable, and stealthy backdoor triggers for large language models (LLMs). The proposed attacks can automatically generate backdoor triggers during the alignment stage of LLMs, and are robust to perturbations. Overall, this work underscores the potential risks associated with LLM alignment and the necessity for enhanced security measures to mitigate adversarial threats in this domain.

strengths:
This paper focuses on an important research problem regarding the backdoor attacks in the LLMs alignment stage. From the attacker's perspective, the authors propose an attack approach named AdvBDGen that has been proven effective in the empirical experiments in the following aspects:
1. The injection of the triggers is easier with just 3% of the fine-tuning data.
2. The injected triggers are more stable compared to previous approaches.
3. The injected triggers are stealthy, making them hard to defend against. 

The research raises relevant security risks regarding the LLMs alignment data.

weaknesses:
1. The overall writing and presentation of this paper should be largely improved. For the current version, it's hard to get an overall understanding of the proposed attack in the abstract and introduction (i.e., how and why the method can work, and the key motivation). The figure 1 doesn't provide too much useful information. Some concrete examples of poisoned samples can be better. 
2. The key motivation of this work and the novelty of the proposed approach are not very clear to me. To the best of my knowledge, semantic-based backdoor attacks are extensively proposed in previous research, including but not limited to the following work: [1,2,3]. However, this paper only mentions "constant triggers" during the motivation and development of the method. This makes the overall contribution and novelty very confusing.  Correspondingly, the relevant semantic-based backdoor attack methods are not considered and compared with in the experiments, making the results not solid enough. 
3. I appreciate the work done by the authors in Sec 4.4 that uses safety training to measure the strength of the proposed attack method. However, this evaluation is far from extensive. More evaluation beyond the simple discussion, including different defense methods and even human annotations should be conducted to measure the strength, stealth, and adaptability of the method. 

[1] Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger. Fanchao Qi et al.

[2] Rethinking Stealthiness of Backdoor Attack against NLP Models. Wenkai Yang et al

[3] Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer. Fanchao Qi et al

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1730665880237

thread_type:
official_review

<thread_object>
creator: reviewer_pafe

<content>
summary:
The paper explores vulnerabilities in aligning LLMs using RLHF, specifically focusing on backdoor poisoning attacks. Traditional backdoor triggers are often fixed and detectable, making them less effective. To address these limitations, the authors propose AdvBDGen, an adversarially fortified generative framework that creates prompt-specific, adaptable, and stealthy backdoor triggers resistant to common defensive measures. This system comprises a generator and two discriminators (strong and weak), which collaborate to optimize the generation of stealthy, effective triggers while maintaining semantic integrity and resisting detection.

The main contributions include:
1. Introduction of AdvBDGen, which uses prompt-specific triggers with varied presentations to bypass conventional defenses.
Demonstration that these generated triggers are highly effective, transferable across models, and robust to semantic perturbations, unlike traditional fixed triggers.
2. Evidence that adversarially trained paraphrased triggers outperform naive paraphrases as reliable backdoors.
3. Highlighting the increased difficulty in detecting and removing these adaptable backdoor triggers, underlining the need for more advanced defense mechanisms in LLM alignment.

strengths:
1. Unlike prior works that use fixed triggers, the paper leverages GAN-style training to explore context-adaptive and paraphrase-based backdoors that are harder to detect and neutralize.
2. The comparative analysis between naive LLM-generated paraphrases and those produced by AdvBDGen underscores the method's efficacy and robustness.
3. The work covers various dimensions of stealthiness and transferability, showcasing how the proposed backdoors withstand semantic perturbations and transfer across different models, enhancing the reliability of the results.
4. By demonstrating that minimal data can be leveraged to install potent and stealthy backdoors, the work underlines the potential real-world risks associated with LLM deployment and training, urging the research community to prioritize defensive measures.

weaknesses:
1. The authors assert that constant triggers can be easily detected; however, they did not provide experimental evidence to support this claim. Notably, constant triggers have shown higher attack performance even at low poisoning rates, which are typically more challenging to detect. This raises the question of whether the proposed method, which involves complex backdoor generation steps, is truly necessary. I recommend providing additional experimental evidence to justify that the benefits of the proposed method outweigh the increased complexity. This will help strengthen the case for its practical advantages.
2. The generator and discriminator training process, which forms the core of your approach, lacks clarity. Specifically, as mentioned in line 237, the generator and discriminator are trained simultaneously. However, for training the discriminator effectively, original, "good," and "bad" prompts are needed, and these prompts are produced by the generator. At the initial stage of training, the generator is expected to create prompts that closely resemble the original, offering minimal informative cues for the discriminator to learn from. This setup implies that successful generator training hinges on the presence of a well-trained discriminator, creating a circular dependency between the two. While GANs face similar challenges, they begin with generators producing nonsensical images that can still be distinguished from real images by the discriminator. In your case, how does an untrained, "cold-start" discriminator differentiate between original and backdoored prompts? This raises concerns about potential model collapse, a known issue in GAN training. However, it appears that the authors did not report encountering this problem. 
    - Can you provide a detailed, step-by-step description of the initial stages of training, focusing on how they handle the cold-start problem for both the generator and discriminator.
    - Can you provide empirical evidence or metrics that demonstrate the stability of your training process over time.
    - Can you discuss any failed approaches or challenges you encountered during the development of this training process and how you overcame them.

3. The defense strategy examined in this work appears rather weak. Since the authors' goal was to train a LLM to produce helpful responses, a straightforward approach would involve using a classifier to detect and filter out misaligned (harmful) responses introduced through data poisoning. This method, which is already implemented on platforms such as [OpenAPI](https://platform.openai.com/docs/guides/moderation), enables model developers to sanitize the training data and mitigate alignment issues effectively. In addition, one can paraphrase the prompt using paraphrasing models or round-trip translation models to neutralize the potential backdoor triggers [1,2]. I would like to see a discussion of the effectiveness of the proposed backdoor triggers against models protected by the defense methods above.
4. The novel is limited, as the use of GAN-style methods for generating backdoor triggers is well-established and has been extensively explored by the NLP and computer vision (CV) research communities [3, 4, 5].


References:
1. Qi, Fanchao, et al. "Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger." ACL 2021.
2. He, Xuanli, et al. "Transferring Troubles: Cross-Lingual Transferability of Backdoor Attacks in LLMs with Instruction Tuning." arXiv preprint arXiv:2404.19597 (2024).
3. Muñoz-González, Luis, et al. "Poisoning attacks with generative adversarial nets." arXiv preprint arXiv:1906.07773 (2019).
4. Saha, Aniruddha, Akshayvarun Subramanya, and Hamed Pirsiavash. "Hidden trigger backdoor attacks." AAAI 2020.
5. Qi, Fanchao, et al. "Turn the Combination Lock: Learnable Textual Backdoor Attacks via Word Substitution." ACL 2021.

questions:
Typos:
1. In figure 1, semnatic-> semantic
</content>
</thread_object>

</thread>
<thread>
creation date:
1730683635239

thread_type:
official_review

<thread_object>
creator: reviewer_aqyp

<content>
summary:
This paper presents AdvBDGen, a generative framework designed to create adaptive and stealthy backdoor triggers in LLMs by leveraging an adversarially fortified training setup. AdvBDGen improves the stealth and resilience of backdoor triggers compared to traditional fixed-pattern approaches by generating prompt-specific, paraphrased triggers. These backdoors are crafted through an adversarial setting involving a generator and two discriminators—a strong and a weak one—that work together to ensure the backdoors remain both effective and undetectable by less sophisticated defenses. The primary contributions of the paper are:

- AdvBDGen introduces a new backdoor generation framework that generates prompt-specific, adaptive triggers for LLMs.

- The backdoor triggers are shown to be effective across different LLM architectures, enhancing their generalizability.

- Highlights the challenge of defense against fuzzy backdoors.

strengths:
- Originality

This paper proposes a novel approach for generating adaptive, prompt-specific backdoor triggers that are context-aware and resistant to traditional defense mechanisms. By using adversarial training with dual discriminators (weak and strong), the framework encourages the generation of subtle, complex triggers that evade detection more effectively than fixed-pattern backdoors.


- Quality

The paper presents rigorous experiments, evaluating the generated backdoors’ effectiveness and resilience. AdvBDGen is benchmarked against standard trigger models, demonstrating its robustness across different poisoning percentages and showing that the generated backdoors are transferable across models.

- Clarity

The paper is well-organized and provides clear explanations of its methodology.

- Significance

The framework addresses an emerging threat in the alignment of LLMs. Given the increasing reliance on preference alignment to ensure safe and aligned AI behavior, the work highlights significant vulnerabilities. By presenting a robust approach to backdoor generation, the paper calls attention to the need for improved defense mechanisms, making it a valuable contribution to AI safety research.

weaknesses:
- Stylistic [1] and Syntactic [2] backdoor triggers can serve as a competing baseline method.

[1] Fanchao Qi et al. Mind the style of text! adversarial and backdoor attacks based on text style transfer.

[2] Fanchao Qi et al. Hidden killer: Invisible textual backdoor attacks with syntactic trigger.

questions:
- Two related works [1,2] are not discussed in this paper. A more detailed discussion on the differences would help readers understand the unique contributions and contextualize this approach within the existing literature.

- Since the generator aims at creating more sophisticated triggers that are subtle enough to bypass the weaker discriminator while still identifiable by the stronger one, should we expect an increase in the prediction error of the weaker discriminator in Figure 2?


[1] Jiahao Yu et al. GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts.

[2] Jiongxiao Wang et al. RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models
</content>
</thread_object>

</thread>
<thread>
creation date:
1732494005921

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Official Comment (1/2)

comment:
We thank all reviewers for their valuable feedback and insightful questions. We are encouraged that the reviewers find the transferability, stealth, robustness, and efficacy of our proposed backdoor attack interesting (paFE, zHca, aqYP). We believe we have addressed all reviewer's concerns in the separate responses. Below we summarize our responses and clarify some common confusion.

---
## Common concerns

### Existing works of semantic backdoors in NLP litreature. 


Reviewers have expressed concerns about the presence of style-based backdoor triggers in the NLP literature. However, prior work in this area has primarily focused on simpler sequence classification tasks and relied on fixed style transfer networks to generate backdoors. In our work, we demonstrate that these fixed style backdoors are not necessarily effective in more complex text generation tasks, such as those requiring contextual response generation (e.g., LLM alignment), as evidenced by our "Paraphrased BD" baseline.

Additionally, existing learnable backdoors in the NLP literature are limited in scope, often relying on simple synonym-based word swapping while adhering to structural constraints, which restricts their applicability. In contrast, our proposed method addresses these limitations by introducing a learnable backdoor capable of integrating semantic objectives (e.g., specific styles or semantics). This approach ensures both efficacy as a backdoor and greater versatility.

### Contribution in terms of architectural design

Some reviewers suggested that our method represents a naive application of a generator-discriminator setup. However, a purely naive approach, as demonstrated in our results, would only yield constant backdoor triggers. In contrast, our method leverages the difference in learning capabilities of weak and strong language models to enhance the discriminator's feedback. To the best of our knowledge, this is the first work to employ weak and strong language models in this manner within a generator-discriminator framework.

This novel formulation enables the generation of input-specific and customizable backdoor triggers, addressing the limitations of prior approaches.

### Lack of Defenses


Some reviewers suggested exploring additional defenses from the NLP literature. However, certain defenses mentioned by reviewers (e.g., RAP) are designed for sequence classification tasks, relying explicitly on label probabilities, and are therefore not applicable to the LLM alignment task addressed in our work. For this reason, we omitted those defenses from consideration.

In response to the reviewers' feedback, we have expanded our evaluation by including **three additional defenses**: ***perplexity-based filtering***, ***n-gram filtering, and round-trip translation***. These complement the four defenses already covered in the paper, which are commonly applied in recent text generation tasks. This brings the ***total number of evaluated defenses to seven***.

Our results ***demonstrate that the proposed attack remains effective against these defenses, underscoring its robustness***. Specifically, the triggers we propose are not only harder to detect but also more challenging to remove once embedded in a backdoored model.

---
</content>
</thread_object>

</thread>


Metareview:
metareview:
After carefully reading the paper, the reviews, the authors' rebuttal, and the additional experiments provided, the meta-reviewer appreciates the significant effort the authors have put into addressing reviewer concerns. The added information and experiments meaningfully enhance the paper, and the improvements in clarity and depth are acknowledged.

However, there remain key concerns that are not fully resolved:

Limited scope of experimentation: While the authors provide detailed experiments on the PKU Beavertails dataset, this singular focus limits the generalizability and scientific impact of the results. Reviewer comments highlighted the importance of validating the proposed method on more popular and diverse datasets, which was not addressed in the rebuttal. Without broader experimentation, it is difficult to assess the robustness and practical utility of the proposed defense method.

Integration of RAP defense: The reviewer appreciate the additional experiments on perplexity-based and round-trip based defense, which significantly improves the paper. However, the reviewers specifically pointed out that integrating the RAP defense method would be a natural and valuable extension to the studied defense with only minor modifications. This could have strengthened the contribution of the paper. However, the authors did not incorporate this suggestion, leaving a gap in the comprehensiveness of the study.

While the paper presents interesting ideas and makes some progress, these unresolved issues, particularly the limited experimental validation, diminish the paper's suitability for publication at this time. Expanding the experiments and addressing the reviewers' concerns in a more thorough manner would significantly improve the paper's impact.