ICML 2025

AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs
Anselm Paulus * 1 2 Arman Zharmagambetov * 3 Chuan Guo 3 Brandon Amos 3 † Yuandong Tian 3 †
Abstract
Large Language Models (LLMs) are vulnerable
to jailbreaking attacks that lead to generation of
inappropriate or harmful content. Manual red-
teaming requires a time-consuming search for ad-
versarial prompts, whereas automatic adversar-
ial prompt generation often leads to semantically
meaningless attacks that do not scale well. In this
paper, we present a novel method that uses an-
other LLM, called AdvPrompter, to generate
human-readable adversarial prompts in seconds.
AdvPrompter, which is trained using an alter-
nating optimization algorithm, generates suffixes
that veil the input instruction without changing its
meaning, such that the TargetLLM is lured to
give a harmful response. Experimental results on
popular open source TargetLLMs show highly
competitive results on the AdvBench and Harm-
Bench datasets, that also transfer to closed-source
black-box LLMs. We also show that training on
adversarial suffixes generated by AdvPrompter
is a promising strategy for improving the ro-
bustness of LLMs to jailbreaking attacks. Our
code is available at http://github.com/
facebookresearch/advprompter.
1. Introduction
Large Language Models (LLMs) are capable of generating
fluent, natural language dialogue and can follow complex in-
structions (Zheng et al., 2023b; Touvron et al., 2023; Dubey
et al., 2024; OpenAI et al., 2024; Jiang et al., 2023; Mistral-
Large, 2024), but also tend to replicate toxic behavior and
generate content that could be considered inappropriate, of-
fensive or harmful (Ganguli et al., 2022; Zou et al., 2023).
To mitigate this issue, most LLMs undergo a process called
safety-alignment, where the model is fine-tuned with hu-
man preferences that reflect positive societal values, leading
*Equal contribution ,†Equal advising 1University of T¨ubingen
2Work done at Meta 3FAIR at Meta. Correspondenc...