Endless Jailbreaks with Bijection Learning

Authors: Brian R.Y. Huang, Maximilian Li, Leonard Tang
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: jailbreaking, redteaming, AI safety, AI alignment, adversarial robustness, adversarial attacks

Abstract:
Despite extensive safety measures, LLMs are vulnerable to adversarial inputs, or jailbreaks, which can elicit unsafe behaviors. In this work, we introduce bijection learning, a powerful attack algorithm which automatically fuzzes LLMs for safety vulnerabilities using randomly-generated encodings whose complexity can be tightly controlled. We leverage in-context learning to teach models bijective encodings, pass encoded queries to the model to bypass built-in safety mechanisms, and finally decode responses back into English. Our attack is extremely effective on a wide range of frontier language models. By controlling complexity parameters such as number of key-value mappings in the encodings, we find a close relationship between the capability level of the attacked LLM and the average complexity of the most effective bijection attacks. Our work highlights that new vulnerabilities in frontier models can emerge with scale: more capable models are more severely jailbroken by bijection attacks.

Decision: accept (poster)
Reviewer scores: 6, 3, 8, 8

Reviews and discussion:
<thread>
creation date:
1730469635242

thread_type:
official_review

<thread_object>
creator: reviewer_snn1

<content>
summary:
This paper introduces an adversarial attack called "bijection learning" that exploits the in-context capabilities of large language models (LLMs) by teaching it a simple cipher to bypass safety guardrails. By teaching LLMs an arbitrary string-string mapping, the attack encodes harmful queries in a "bijection language" which can easily be decoded back to English. The approach is effective across various models, with the attack's potency increasing with model capabilities. The authors demonstrate this with extensive experiments, showing high Attack Success Rates (ASR) on a range of safety-trained frontier models such as Claude and GPT-4o. The paper argues that as models become more capable, they become more susceptible to cipher attacks like bijection learning, posing significant safety challenges.

strengths:
- **Simple and scalable attack that exploits in-context learning in LLMs**: The bijection learning attack method, allows you to sample a large number of text mappings that transform the input text into a bijection language. This allows you to sample them until you find a successful attack, which is a powerful red-teaming scheme.
- **High ASR on frontier models**: The attack achieves an ASR of 94% on Claude 3 Opus, which is impressive.
- **In-depth analysis of how effective the attack is with different scales**: The authors find that the attacks are stronger with scale. Smaller models fail to learn bijections, but the attack can be tuned for difficulty by changing the “fixed size” to work on less capable LLMs.
- **Contributions to Safety Research**: The paper identifies a new risk factor in AI safety that scales with model capabilities, emphasising the dual-use nature of advanced reasoning in LLMs. It underscores the necessity for evolving safety mechanisms that match these capabilities, providing crucial insights for AI developers on robust mitigation strategies.

weaknesses:
- **Claims need to be better explained or backed up with a citation:**
    - “Scale of models and safety is under explored”. I’m not sure this is true because most jailbreak attack papers text over various scales. MSJ, Jailbroken, etc, all look at this (or they at least look at how safety changes with scale). You need to make this claim more specific because I agree that using the improved capability to attack via ciphers is underexplored.
    - “Model capability increasing also widens the attack surface”. I think this is unclear. It does introduce new novel vulnerabilities, but it could also be the case that the total attack surface shrinks even when there are these new cipher-based vulnerabilities. So again, I would make this claim more specific and less general (unless you have data to back it up or can cite a paper that does show this generally).
    - Is your method for jailbreaking novel? Ciphers are not new, but perhaps your version of bijection learning in context is novel. I think it is worth being clearer on what is novel and also doing a better job at comparing and contrasting in the related work section.
    - Is the approach scale agnostic? Perhaps to a certain point but this would break down? What is the smallest LLM you tried the approach on? You say “new language model risk factors may emerge with scale” but also say the approach is “scale agnostic”. I think making the story clearer and not mixing the two here would be good. Also, it is important not to confuse bigger models with improved alignment fine-tuning, and it can be tricky to make claims about this since labs do not release details of their training.
    - “while many other jailbreak methods only work on smaller models” - can you cite this or explain more? Most jailbreak techniques, e.g., GCG, PAIR, TAP, etc, work well across all model sizes, so I am not sure this claim is true.
    - “more powerful on more capable models” - can you quantify this?
    - “arguable the safest frontier model at time of writing” - can you cite this result or remove it? Gemini 1.5 Pro is very competitive and I’m not sure which is ultimately better.
    - “endless mappings that yield successful jailbreaks” - how many templates did you test, and how many of them worked? It would be good to quantify this in your contributions.
    - “certain complexities of bijection learning systematically produce certain failure models, giving further insight into our scaling laws” - what is the complexity of bijection learning? What are the failure modes? Can you give a one-sentence summary to help ground this claim?
    - “more severe harmful behaviors more frequently” - how much more? Can you give an idea of the jump from 3.5-level models to 4? How do you know that it is bijection learning that induces more harmful behaviour? It could simply be because the model is more capable. Perhaps compare “egregiousness” with other jailbreaks and see if bijection learning induces more harmful ones. Otherwise, this claim isn’t interesting.
    - Say more about your scaling laws in the contribution section - do they allow you to forecast future capabilities? What equation are you fitting?
    - “harmful model responses produced by our attack are thus fully reproducible” - have you tested this on all frontier models? Even at temp 0, output diversity exists, so be careful of your claim here.
    - “Our work is the first to produce attacks that are simultaneously black-box, automated, and scale agnostic” - I don’t think this is the case. PAIR and TAP are prime examples of methods that fit these criteria.
    - In the results section: “[whitebox methods] fail to transfer to the newest” - could you cite this? There is some GCG transfer. If you have numbers, then include them in the paper (even if it is in the Appendix)
- **Some discussion points can be improved:**
    - I think you can motivate your work better e.g. many attacks are caught by defenses such as output classifiers, but your work can bypass these easily by using a cipher that the output classifier won’t understand. However, it is unclear if you use an input+output classifier if they will catch your attacks or not. The classifier must be at the same capability level as the generation model.
    - The discussion of the “bijection jailbreaks can be universal” can be improved by making the point that the algorithm itself leads to a universal attack and relies on a “fuzzing” technique that samples a prompt template until one works. See LLM fuzzer paper https://www.usenix.org/conference/usenixsecurity24/presentation/yu-jiahao.
- **Related work lacks comparing and contrasting with their work:**
    - Safety of LLMs - this does not compare and contrast with your work. Is it even relevant
    - Adversarial attacks against LLMs - please compare and contrast more. Maybe separate the cipher work into its own section and contrast it in more fine-grain detail.
    - Adversarial robustness in LLMs - is this relevant to your work since you don’t look into defenses?
- **Lack of threat model:** Why are you working on black-box attacks on frontier models? Why is this more impactful to work on than white-box attacks? (I think it is)
- **Improving method explanation**:
    - Desiderata section. Universal and/or automated - this sentence is hard to parse, perhaps separate into two bullets
    - Fixed size (section 2.2) - this bullet point makes it hard to understand what you mean. It is a simple concept, and I think it could be explained more clearly and with fewer words. I think it would be easier to define complexity, C, as the number of characters that map to something else. Then it is easy to understand that C=0 means there is no change, and C=26 means you change each letter for something else.
    - The false positive rate is not measured. Report the false positive rate when you talk about it. Also, do you check every single response in all your experiments? Some more clarity on your method here (including a rubric your human evaluators follow) would be good as it impacts the ASRs throughout the paper a lot and will help people replicate.
    - Add how you filter HarmBench, e.g. do you just use standard direct requests and remove copyright/context ones?
- **Lack of black-box baselines and explanation of baselines used**:
    - There should be more baselines, e.g. vs PAIR and TAP. You could evaluate these with the BoN methodology too. I expect PAIR to get similar ASRs on GPT4o-mini. Without these, it makes
    - Please explain your implementation for ascii, caeser-cipher, morse code and self-cipher. (I suggest significantly cutting down section 5 to make room). The reader is left guessing how these work. Is it fair to compare them when you have a different attack budget for bijection learning?
- **The presentation of results is poor as the figures are too small and contain too much data.** In general, I’d recommend thinking about the main insight you want the reader to take away from each figure and majorly simplifying it so that is the case.
    - Figure 3 - this is a little messy. Why do we need the tables on the right when the bar charts have all the details? I think it would be great to have a bar chart with the best-fixed size for each bijection type and compare directly against baselines for each model (using the full 320 behaviors since HarmBench-35 isn’t large enough). Then, a separate figure that ablates the fixed size. Then, it makes it easy to highlight the insights you’d like to convey in the caption and main text. What is the attack budget for these?
    - I think Table 1 contains the exciting results. I’d suggest leading with this before Figure 3. I’d also like to see comparable PAIR or TAP ASRs on the HarmBench 320 and AdvBench 50 set in the table. I think it is less important to show the fixed size and attack budget and just show the ASRs vs the baselines. Also, maybe fix the attack budget to the same as the budget for PAIR so it is more directly comparable. Also, how did you choose the current attack budgets? Was that the plateau point?
    - Figure 4 left - This is an awesome plot, and I think you should make a bigger deal out of it.
    - Figure 5 - I think this could be better as a line plot. Perhaps choose to plot either ‘digit’ or ‘letter’ rather than both to simplify. Share legend for all. Also, I think just having three lines: successful attack, refusal, and unhelpful non refusal would help simplify this. Potentially, just plotting unhelpful non-refusal for each model would get your point that smaller models can’t learn complexity bijections the best.
    - Figure 6 - share the legend so each plot is bigger. The main point you are trying to make is that the capability degrades as fixed size increases, but this message is easily lost with the amount of information you present. I’d recommend simplifying - just show it averaged over models and maybe drop some of the multi-digit results. You can put this plot in the appendix to refer to some nitty gritty details.
    - Figure 7 - I don’t think this is a scaling law. It is more of a Pareto frontier plot where you want to show the trade-off in two metrics. A scaling law would involve the amount of computing, data size, or size of the model. Did you experiment with different fits? A quadratic might not be as good as a higher-order polynomial. Why is a quadratic the right fit for this data?
    - General plotting advice: Please use error bars for all plots that take into account the sample size used (this can just be a standard error of a proportion that doesn’t involve rerunning many seeds). Use a colour-blind pallet, and make the font in figures a similar size as the font of paper.

questions:
Most of my questions are in the weaknesses section, but here are some other questions/discussion points.

- What does “endless” mean? How is it different from any “adaptive” attack (like PAIR or random search suffixes) that will craft many potential jailbreaks until it finds one that works? I’m not sure emphasising this makes sense. Maybe you reframe it as an “Adaptive in context learning attack”?
- In the abstract, what does universal mean? Is it transferred across harmful requests or across models? Do you have a % of how the same bijection template transfers across requests and across models?
- Multi-turn conversation history - do you think it would be best to use the term few-shot learning here and then, in methodology, talk about how many shots you use (it looks like it is 10-shot?). Also, you call it “teaching examples” later. I would use few-shot since it is clear what it means.
- What defenses work / don’t work against your attack?
- Intro: I wouldn’t describe perplexity filtering, paraphrasing, etc as an alignment technique like RLHF. I’d describe it as an adversarial defense against jailbreaks.
- Maybe add the GPT-4o rewriter to correct typos in Figure 1, so it is clear how it works without needing to read the paper.

This paper has the potential for a much higher rating, but not in its current form. I would happily raise my score if the claims I mention in the weaknesses section are better explained and the results are significantly simplified in the figures and presented well. In addition, I’d like to see the comparison to adaptive black-box attack baselines like PAIR with a comparable attack budget.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730509483667

thread_type:
official_review

<thread_object>
creator: reviewer_53mm

<content>
summary:
This paper proposes a new method for jailbreaking LLMs by means of communicating in an encrypted format.
The LLM is first given a simple encryption algorithm for encrypting and decrypting strings.
Then, the user and LLM communicate for several rounds in this encrypted format.
After some time, the user will ask a malicious query (e.g., "Design a weapon") in this encrypted format; this paper finds that the LLM response, when decrypted, is often a jailbroken one.
This style of attack, dubbed a "bijection attack", is more successful against stronger LLMs.

strengths:
The authors present an interesting attack that demonstrates the surprising ability to scale with LLM power: stronger LLMs appear more susceptible to this attack. Moreover, the proposed attack achieves key desiderata of jailbreaks: against a black-box target, universal/automatable, and scalable. These qualities make "bijection attacks" a valuable benchmark for LLM developers to consider when evaluating safety.

weaknesses:
In my opinion, this paper does not have clear weaknesses. However, given the state of jailbreaking research, I do not think that this style of attack paper is scientifically or technically exciting. To change my opinion, I would like to see some deeper technical insights + experiments, possibly with the authors' proposed defense strategies in Section 5 --- but this may be unreasonably ambitious in the rebuttal time frame. While my impression leans on the negative side, I am okay with accepting this work if the other reviewers do not have strong objections.

questions:
It would be good to see additional discussion on how LLM developers might defend against this style of attack.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730617939937

thread_type:
official_review

<thread_object>
creator: reviewer_wl23

<content>
summary:
This paper proposes a general framework for jailbreaking attacks with bijection encoding ciphers. Experiments show that the attack is effective on a wide range of frontier language models. They also find out that the bijection cipher attack is more effective on larger and stronger language models.

strengths:
1. ASR of the proposed method is high on many frontier LLMs.
2. The authors did comprehensive experiments to verify the effectiveness of their method. The results reported in Section 3.3 is interesting.

weaknesses:
1. The novelty of this work is questionable given many existing cipher based jailbreaking attacks. It seems the only difference between this paper and existing works is that this paper proposes to use a system message to customize general cipher encodings. I'm not confident about whether the contributions are enough for ICLR.
2. Comparisons between many other cipher-based jailbreaking attacks are missing, including but not limited to:
[1] When “Competency” in Reasoning Opens the Door to Vulnerability: Jailbreaking LLMs via Novel Complex Ciphers
[2] Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characters

questions:
Please see the weakness part above. Could the authors explain more on how their proposed methods differ from the existing ones?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730668520492

thread_type:
official_review

<thread_object>
creator: reviewer_czkw

<content>
summary:
The paper discusses an approach to jailbreaking via bijection learning. Specifically, they generate a random transformation over characters that shifts the input prompt out of the safety fine-tuning distribution. They find that models prompted to answer inputs in bijection format are more likely to output harmful content compared to standard model inference.

strengths:
1. The algorithm is clear, simple, and admits random sampling for endless bijections. 
2. The analysis is comprehensive. I really appreciated the scaling analyses for 1) the n in best-of-n and 2) the ASR vs model capabilities frontier showing that more capable models may be more susceptible.

weaknesses:
1. The main ideas presented in this paper have been identified in prior works under mismatched generalization [1] and lack of robustness to out-of-fine-tuning distribution prompts such as low-resource languages [2,3]. [1] also makes the observation that transformation-based jailbreaks benefit from increasing model scale. As such, this paper extends these ideas rather than introduces them. 
2. I believe the comparison in Figure 3 to the baselines is not apples-to-apples since bijection learning performs a best-of-6 while the baselines are effectively best-of-1. I think a more fair comparison is either doing best-of-1, or having a combined baseline that's an ensemble of six reasonable baselines.

[1] https://arxiv.org/abs/2307.02483
[2] https://arxiv.org/abs/2310.02446
[3] https://arxiv.org/abs/2309.10105

questions:
1. "We search for universal attack mapping candidates from our earlier evaluations by selecting attacks which produce a strong response for a particularly malicious intent. For a selected mapping, we generate the fixed bijection learning prompt and evaluate it as a universal attack on HarmBench" (quote starting from line 342). I was confused by this phrasing, is the scaling plot in Figure 4 for best-of-n at n=1 done for a single random sample of a language, or is it done for the single best language subselected from all the prior experiments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732279272478

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to all reviewers

comment:
We thank all the reviewers for their thoughtful and helpful insights. We have made significant revisions to the paper based on feedback and encourage all reviewers to revisit the revised version of the paper, especially the updated Table 1 with comparisons to baselines.

Here, we discuss a few shared concerns between reviewers.

**Novelty claims**. We agree that we could do more to differentiate our method from prior work and make the case that our contributions are novel and significant. We rewrite our related works and contributions sections (1.1 to 1.3) and refer reviewers to these sections for further context. However, we also think it’d be useful to summarize our case here for further discussion.

Our attack is “universal,” which means an attacker can use it by inserting their prompt into a template using a simple procedure (encoding the attack intent in bijection language) that does not require any human or AI assistance. Most jailbreaks are not universal. For example, TAP requires a significant feedback loop between an attacker model and the target model that greatly diminishes its ease of use compared to a universal jailbreak.

Previous encoding-based attacks encode a harmful prompt in some particular encoding, like Base64, ASCII, or ROT13 that the model already knows about through training. Compared to these works:
1. We are the first to leverage models’ ability to learn encodings in-context, not just those a model has memorized.
2. Our sampling method is a powerful "fuzzing" technique that allows us to perform best-of-n sampling from an essentially endless pool of possible encodings, which doubles our ASR compared to using only a single random encoding. Best-of-n sampling is not possible using previous techniques – an attacker can keep trying different well-known ciphers, languages, etc. that have been proposed in other papers, but will eventually run out of options.
3. We use quantitative parameters to scale the encoding difficulty to best match the model’s capability level. In particular, our dispersion parameter d, 0<=d<=26, controls how many different letters are mapped to an encoded sequence, where d=0 is plain text and d=26 permutes all letters. Previous attacks are rather blunt: either the model knows the selected encoding or it doesn’t. However, for bijection learning, if a certain model cannot learn to permute all the characters, we can tone down the dispersion to map only a few letters to something else. Hence, our method is uniquely scale adaptive in a way that other methods are not.
4. Our capabilities vs ASR analysis proposes a new reason why our attack works, which is that translating bijections is a hard enough task that it saps the model’s capabilities overall and in particular steals capacity away from safety mechanisms. This is different from the analysis from previous work of “safety tuning is in English, so it doesn’t generalize to other languages” and exposes a potentially new direction in jailbreaking.
5. Finally, our empirical results are interesting, since we achieve extremely high ASRs on frontier models compared to previous methods. See next point.

**Better comparison to baselines**. Reviewer WL23 noted that we should add more encoding baselines. Reviewers CzKw and sNn1 noted that it may not be fair to compare best-of-n to baselines that only create one attack prompt.

Best-of-n sampling is only possible with our method and is one of its main advantages. However, we agree that it would be interesting if we artificially gave previous work this degree of freedom. As suggested, we ensemble 11 of the most popular and best performant encoding baselines (7 more than before) and consider this ensemble attack successful for an attack intent if any one attack succeeds, similar to best-of-11 sampling. **Bijection learning still outperforms by over 30 percentage points on all models** (see Table 1). In fact, best-of-1 bijection learning still outperforms this ensemble on Claude models, and best-of-5 bijection learning outperforms on all models (see Table 3).

Reviewers also suggested we compare to PAIR and TAP, which we add to Table 1. Again, we outperform by over 25 percentage points on all models when using the same (or larger, in the case of TAP) attack budget with these attacks compared to our budget for bijection learning.

**(New) Defense techniques**. We add an analysis of a defense strategy against bijection learning in Section 5: a guard model defense that performs safety classification on model inputs and outputs, nullifying the output when unsafe behavior is detected. The input filter takes the entire context, and the output takes the entire context plus the model response. Curiously, even when the guard model is the same as the target model (Claude 3.5 Sonnet), the ASR of bijection learning is only reduced by 10 percentage points, so even “safety-capability parity” [1] may not be sufficient to defend models from bijection learning. 

[1] https://arxiv.org/abs/2307.02483
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a novel bijection method leveraging side-channel techniques to jailbreak large language models (LLMs). The findings indicate that stronger LLMs are more vulnerable to jailbreak attacks and demonstrate state-of-the-art (SOTA) attack success rates. The proposed method also achieves high success rates in jailbreaking Claude, a leading LLM. These claims are supported by experimental results and comparisons to baseline attack techniques.

The strengths of the paper include its highly effective method for jailbreaking LLMs, which outperforms existing approaches in terms of success rate. The methodology is rigorously evaluated, demonstrating robustness across multiple LLMs, including Claude. This highlights the practical implications and potential risks of deploying strong LLMs in sensitive environments. The paper also finds that that stronger models are easier to jailbreak.

The weaknesses of the paper lie in its lack of novelty and limited scientific contribution to advancing the understanding of LLM vulnerabilities. While the bijection method is effective, the finding does not offer new insights into the fundamental mechanisms underlying these vulnerabilities. This paper would be useful if this capacity degradation argument was real and substantiated. Unfortunately, in the current state, their argument does not seem fully convincing

The primary reason for accepting the paper shall be its empirical results, which stands out as one of the better jailbreak papers. However, there would also be reasons to reject this paper due to its insufficient contribution to advancing scientific understanding. While the method demonstrates strong empirical results, the findings do not significantly enrich the field's knowledge or provide actionable recommendations for addressing the security challenges posed by jailbreaking attacks.