A Realistic Threat Model for Large Language Model Jailbreaks

Authors: Valentyn Boreiko, Alexander Panfilov, Vaclav Voracek, Matthias Hein, Jonas Geiping
Primary area: foundation or frontier models, including LLMs
Keywords: LLM, jailbreaks, threat model, robustness

Abstract:
A plethora of jailbreaking attacks have been proposed to obtain harmful responses from safety-tuned LLMs. These methods largely succeed in coercing the target output in their original settings, but their attacks vary substantially in fluency and computational effort. In this work, we propose a unified threat model for the principled comparison of these methods. Our threat model combines constraints in perplexity, measuring how far a jailbreak deviates from natural text and computational budget in total FLOPs.
For the former, we build an N-gram language model on 1T tokens, which, unlike model-based perplexity, allows for an LLM-agnostic and inherently interpretable evaluation. We adapt popular attacks to this new, realistic threat model, with which we, for the first time, benchmark these attacks on equal footing. After a rigorous comparison, we find attack success rates against safety-tuned modern models to be lower than previously presented and that attacks based on discrete optimization significantly outperform recent LLM-based attacks. Being inherently interpretable, our threat model allows for a comprehensive analysis and comparison of jailbreak attacks. We find that effective attacks exploit and abuse infrequent N-grams, either selecting N-grams absent from real-world text or rare ones, e.g., specific to code datasets.

Decision: reject
Reviewer scores: 5, 5, 3, 8

Reviews and discussion:
<thread>
creation date:
1730106147954

thread_type:
official_review

<thread_object>
creator: reviewer_m1dj

<content>
summary:
This paper introduces a unified threat model for evaluating jailbreak attacks on safety-tuned LLMs. Recognizing the multitude of existing jailbreak methods that vary in success rates, fluency, and computational effort, the authors propose a framework that combines constraints on perplexity—measuring deviation from natural text—and computational budget quantified by FLOPs. To achieve an LLM-agnostic and interpretable evaluation, they construct an N-gram model based on one trillion tokens. Adapting popular attacks within this new threat model, the paper benchmarks these methods against modern safety-tuned models on equal footing. The findings indicate that attack success rates are lower than previously reported, with discrete optimization-based attacks outperforming recent LLM-based methods. Furthermore, effective attacks tend to exploit infrequent N-grams, selecting sequences that are either absent from real-world text or rare, such as those specific to code datasets.

strengths:
* Unified Threat Model: The paper addresses the critical need for a standardized framework to compare various jailbreak attacks, providing clarity in a field crowded with disparate methods.

* Interpretability: By employing an N-gram model for perplexity measurement, the threat model remains interpretable and LLM-agnostic, facilitating a deeper understanding of why certain attacks succeed or fail.

* Comprehensive Benchmarking: Adapting and evaluating popular attacks within the proposed threat model allows for a fair and rigorous comparison, advancing the discourse on LLM vulnerabilities.

weaknesses:
* Comparison with Existing Methods: The paper would benefit from a direct comparison with existing perplexity detectors, such as the one proposed by Alon et al. (arXiv:2308.14132). This would contextualize the proposed model within the current state-of-the-art and highlight its relative advantages.

* Perplexity Measurement Limitations: While the N-gram model offers interpretability, it may not capture the nuances of natural language as effectively as model-based perplexity measures, potentially affecting the evaluation's accuracy.

questions:
* Clarify Equation 1: Provide a complete and formal definition of the Judge function. This will enhance the paper's clarity and reproducibility.

* Enhance Comparative Analysis: Include a comparison with existing perplexity detectors, specifically the method proposed in arXiv:2308.14132.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730584271412

thread_type:
official_review

<thread_object>
creator: reviewer_pgg1

<content>
summary:
The paper introduces a new "threat model" for LLM jailbreaks (I am not convinced that this is the right framing here) using an N-gram perplexity approach. There are some interesting ideas, but I think the framing needs adjustment before being ready for publication.

strengths:
- I thought the idea of using the N-gram was interesting.
- The paper is fairly clear written, and the plots are clearly presented.
- I thought the analysis showing an N-gram perplexity constraint increased compute time for GCG interesting, and that it reduces ASR. The analysis comparing ASR against flops was generally very interesting.

weaknesses:
- I'm not sure if the contribution is the threat model or the N-gram language model perplexity filter?
    - As far as I understand, the "threat model" is basically assuming the N-gram approach is the right way of doing things, but I am not sure that is clearly established here? If the point of the paper is to establish the threat model, there should be lots of evidence it is an appropriate defence.
    - I don't find this evidence in the paper. It is simply assumed that this is an appropriate defence?
- I am not convinced the threat model is the best one. I think the best threat model is trying to break what Frontier AI labs have released. I think claiming the threat model here is realistic is significantly overclaiming.
- I think the benefit and results section would benefit from making clear the implications of the results much more.

questions:
- I am generally confused about the "threat model" framing, e.g., "universal threat model for comparing attacks across different models", how does the threat model actually allow you to compare?
- The claim in Table 1 is "The Lack of an LLM-Agnostic Threat Model Renders Attacks Incomparable." But I think the attacks are comparable, just on different axises? I could not follow precisely what the claim is here? I also think the most important thing is ASR? The paper makes the claim many times that the attacks are incomparable, but I just cannot follow this.
    - In terms of needed progress, the threat model is basically what frontier AI labs release? I'm not sure a new threat model is not what is needed.
- How does the N-gram model do on longer user queries? Presumably the perplexity increases substantially with longer queries. Does this mean that this defence would not work well with long-context models? Some of the latest models from Frontier labs can have very long context lengths. This makes me think the threat model might not actually be appropriate.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730661086369

thread_type:
official_review

<thread_object>
creator: reviewer_ygjw

<content>
summary:
Jailbreaking attacks on large language models (LLMs) are widely studied in the literature. But those attack prompts are usually non-natural text. This paper proposes an N-gram method to measure the fluency of generated attack prompts. It shows that this simple approach can filter out several existing jailbreaking attacks and significantly reduce their attack success rates. The paper then proposes an adaptive attack which considers the N-gram of the attack prompt during generation. The results show it can boost the attack performance of existing jailbreaking methods.

strengths:
1. A systematic study on jailbreaking attacks is an important direction and can lay the foundation for future research. This paper provides a good study in this space, which helps the community to better develop new techniques in attacking and protecting LLMs.
2. The proposed N-gram model is effective in filtering several attacks that generate jiberesh text. Those jailbreaking prompts are quite different from natural sentences, causing them easily detectable and hence not robust.
3. The evaluation is comprehensive, including multiple recent LLMs and safely aligned models. The baseline attacks are chosen from the state-of-the-arts.

weaknesses:
1. It is known that several existing jailbreaking attacks generate non-natural text. There have been many proposed methods for filtering such jailbreaking prompts [1][2]. This insight mentioned in the paper is not new. The proposed approach of using the N-gram is straightforward. The novelty hence seems limited.
2. According to Table 2, using the perplexity measure of llama2-7b can distinguish the two optimized suffixes. Why is this approach not used to evaluate jailbreaking attacks in Tables 3 and 4? Additionally, there are many other filtering methods such as [1][2]. It is important to compare the performance of the proposed approach with these techniques.
3. The paper introduces an adaptive attack that considers the N-gram measure during adversarial prompt generation. It is strange that the paper introduces an attack against the proposed measure and then uses the same measure to evaluate the performance. It is similar to self-verifying correctness. It is suggested to use other filtering methods such as [1][2] and the llama perplexity to evaluate the final attack performance.
4. The case shown in Table 2 for the adaptive attack does not seem like natural text as well. Why does the N-gram model not filter this attack prompt? For example, the phrase “A questions their She” very unlikely exists in normal text. With the 8-gram model used in the paper, it should be able to filter out this. Could the authors explain why this case bypass the detection?


[1] Alon, Gabriel, and Michael Kamfonas. "Detecting language model attacks with perplexity." arXiv preprint arXiv:2308.14132 (2023).
[2] Inan, Hakan, et al. "Llama guard: Llm-based input-output safeguard for human-ai conversations." arXiv preprint arXiv:2312.06674 (2023).

questions:
Please see the weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692731010

thread_type:
official_review

<thread_object>
creator: reviewer_atuy

<content>
summary:
This paper proposes a new threat model to compare different attacks. This threat model includes a perplexity filter based on an N-gram language model and constraints on FLOPs. A fine-tuned LLM judge measures the ASR. Many existing attacks fail in this threat model. With the consideration of the proposed perplexity filter, the adapted attacks can restore many ASRs.

strengths:
1. A unified model to evaluate different attacks.
2. N-gram LM has certain advantages.

weaknesses:
1. Only considered white-box attacks. In the real world, black-box attacks are more practical. As shown by Figure 13, white-box attacks in this new threat model have lower transferability. That is, this threat model cannot measure black-box attacks very well.

2. The perplexity filter is not new.

3. There are other defenses such as [instruction filter](https://arxiv.org/abs/2312.06674), and [random perturbation](https://arxiv.org/abs/2310.03684), etc. Why doesn't the threat model consider them?

4. Evidence is needed to show that N-gram LM is better than LLM. Some experiments are necessary.

questions:
Please see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732543799216

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Meta Reply by Authors

comment:
We thank all reviewers for their time and assessment of our work. We appreciate that they found our evaluation “comprehensive” (YgJw, M1dJ), the paper “clearly written” (PGG1), and highlighted the "unified model to evaluate different attacks" as our strength (ATuy).

\
**Comparison with LLM-based filters**

First, we want to clarify the difference between the N-gram language model- and LLM-based filters and why we chose the former - a point that came up in several reviews (ATuy, YgJw, and M1dJ).

We do think that there are a number of advantages to the setup we propose (as discussed above), but to address this question directly, we have now added the suggested comparison to the LLM-based filters:

|                            | **ASR w/o LG2**                     | **ASR w/ LG2 (Input)**        | **ASR w/ LG2 (Input+Output)** |
|----------------------------|-----------------------------------|---------------------------|---------------------------|
| **Baseline**               | 0.68                             | 0.28                      | 0.20                      |
| **Adaptive to N-gram LM PPL (TPR=99.9%)** | 0.46             | 0.32                      | 0.22                      |
| **Adaptive to Self-PPL (TPR=99.9%)**      | 0.44             | 0.30                      | 0.18                      |
| **Adaptive to Llama Guard 2**             | 0.46             | 0.46                      | 0.24                      |

* Here, we reran our entire pipeline using Llama2-7B to measure LLM-based perplexity and, additionally - using Llama Guard 2. For the Llama2-7B (Self-PPL) filter, we calibrate it to a 99.9% TPR rate on benign prompts and run strong adaptive attacks against it using PRS. To our knowledge, adapting PRS to LLM-based defenses using rejection sampling is novel.

* As shown in the table above, the ASR achieved with the N-gram-based filter is 46% versus 44% with the Llama2-7B-based PPL filter and 46% - with the Llama Guard 2. Based on these results, we see that N-gram-based and LLM-based filters have a similar effect so that in terms of ASR, there is no advantage to using the LLM-based filters. However, there are three important advantages of N-gram over LLM-based filters (see Section 1) that make it much better suited for the design of our threat model: i) model-agnostic; ii) negligible compute cost; iii) interpretable and provides insights to LLMs failure modes.

-----

\
**Incorporated suggestions**

We also appreciate the constructive feedback from all reviewers. Thanks to their comments, we have improved the paper as follows:

* Reviewers (ATuy, YgJw, and M1dJ) suggested providing a comparison to the LLM-based perplexity.

  * To incorporate this, we added Section I in the Appendix and discussed it in answers to the reviewers below.
 	
* Reviewers (ATuy, PGG1) suggested discussing the attack's performance when those are evaluated against closed-source models in more detail.

  * We expanded and improved Section H in the Appendix and discussed it in the answers below.

* Per the suggestion of Reviewer PGG1, we provided a clearer description of the implications of the results in Section 5.5. 

* Additionally, we fixed typos and phrasing throughout the paper.	
* Lastly, we understand that Reviewer PGG1 might have a different framing of our approach. Still, we hope we can find common ground regarding the value of our detailed evaluation of a perplexity-based threat model for a number of modern jailbreak attacks and novel adaptive attacks, providing value to the community. 

  While it was pointed out that companies like OpenAI and Anthropic [1] do not use perplexity filters, our results clearly show that even after a stress test (using comprehensive, adaptive attacks), these filters do indeed reduce attack success and increase the attacker’s effort. As such, our work provides strong evidence that these filters should also be deployed by these companies, as already done by other companies such as NVIDIA [2].

-----

\
**References**

[1] https://www.anthropic.com/rsp-updates

[2] https://docs.nvidia.com/nemo/guardrails/user_guides/jailbreak_detection_heuristics/README.html
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a unified threat model for jailbreaks, that comprises attack success but also fluency and computational effort.

I think this paper uses the term "threat model" in a somewhat unusual and confusing way, by referring to it as a "weak version of a defense" (c.f. https://openreview.net/forum?id=1kMTJnqmyl&noteId=Dbv15jJwz2). 
In security research, a threat model is typically a set of constraints or assumptions placed on an attacker (e.g., what are the attacker goals, what invariants would an attack violate, what can the attacker do or not do).

So a threat model might be: the attacker wants to jailbreak a model and has to do so in a way that bypasses a perplexity filter, and uses at most X computation.

But there are two issues I see with this:
(1) the threat model seems directly tied to a very specific way of doings perplexity filtering, namely with an N-gram model. Why not treat this more generally?
(2) placing bounds on an adversary's computational power is tricky, unless we go the cryptographic route and claim an attack is intractable. What if one attack is parallelizable and another isn't? What if one attack has been better optimized than another? Generally I would not make computation part of a threat model unless there's a good reason to claim that an adversary would be strongly computationally bounded.

Overall, while the study of perplexity filters is interesting, I would thus recommend rejection and encourage the authors to clarify what they mean by a "threat model" in this context.