EFFICIENT JAILBREAK ATTACK SEQUENCES ON LARGE LANGUAGE MODELS VIA MULTI-ARMED BANDIT-BASED CONTEXT SWITCHING

Authors: Aditya Ramesh, Shivam Bhardwaj, Aditya Saibewar, Manohar Kaul
Primary area: foundation or frontier models, including LLMs
Keywords: JailBreak, AI Security, LLM Vunlnerability

Abstract:
Content warning: This paper contains examples of harmful language and content.
Recent advances in large language models (LLMs) have made them increasingly vulnerable to jailbreaking attempts, where malicious users manipulate models into generating harmful content. While existing approaches rely on either single-step attacks that trigger immediate safety responses or multi-step methods that inefficiently iterate prompts using other LLMs, we introduce ``Sequence of Context" (SoC) attacks that systematically alter conversational context through strategically crafted context-switching queries (CSQs). We formulate this as a multi-armed bandit (MAB) optimization problem, automatically learning optimal sequences of CSQs that gradually weaken the model's safety boundaries. Our theoretical analysis provides tight bounds on both the expected sequence length until successful jailbreak and the convergence of cumulative rewards. Empirically, our method achieves a 95\% attack success rate, surpassing PAIR by 63.15\%, AutoDAN by 60\%, and ReNeLLM by 50\%. We evaluate our attack across multiple open-source LLMs including Llama and Mistral variants. Our findings highlight critical vulnerabilities in current LLM safeguards and emphasize the need for defenses that consider sequential attack patterns rather than relying solely on static prompt filtering or iterative refinement.

Decision: accept (poster)
Reviewer scores: 6, 8, 5, 6

Reviews and discussion:
<thread>
creation date:
1729157151613

thread_type:
official_review

<thread_object>
creator: reviewer_pzyz

<content>
summary:
This paper proposes a novel method for jailbreaking large language models (LLMs) through "Sequence of Contexts" (SoC) attacks and utilizes a multi-armed bandit (MAB) framework to automate the optimization of the attack process. The paper also provides an in-depth theoretical analysis of the sequence length required for a successful jailbreak and the convergence of total rewards.

strengths:
The study not only experimentally demonstrates the efficiency of the proposed method, achieving an attack success rate of over 95%, but also provides a solid theoretical foundation for LLM jailbreak attacks.

weaknesses:
1. It is a natural question: why do you not compare your work with other methods，such as GCG[1] , AutoDAN[2], PAIR[3], RENELLM[4] and so on? 
2. Since your work requires selecting a sequence of context-switching queries, I am curious about it time complexity. 
3. In my opinion, testing only on Mistral and Llama is not sufficient to demonstrate the advantages of your work. Moreover, you have only chosen small LLMs (up to 8B), which is not convincing. As far as I know, Llama has a 13B version. What's more, CHATGPT is necessary to be choose.
In conclusion, without comparisons, it is difficult for me to fully assess the contributions of this paper, especially considering that there are many papers on LLM jailbreaks. If you address my concerns, I will consider giving a higher score.

Ref:
[1] Universal and transferable adversarial attacks on aligned language models.
[2] Autodan: Generating stealthy jailbreak prompts on aligned large language models.
[3] Jailbreaking black box large language models in twenty queries.
[4] A Wolf in Sheep’s Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily

questions:
See weekness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730116585192

thread_type:
official_review

<thread_object>
creator: reviewer_rsu9

<content>
summary:
This paper discovers the LLMs intend to reject direct malicious queries (DMQs), but answer when followed by content switching queries (CSQs). For automatically generating DMQs and CSQs, the authors propose a framework based multi-armed bandit (MAB) to jailbreak LLM automatically. They introduce a dataset of CSQs based on MAB. And they give a mathematical derivation for the proposed method to prove key bounds.

strengths:
### 1. The discovery about switching the context leads jailbroken
This paper presents a context switching attack, which is a novel method compared to existing works. And combined with multi-armed bandit, SoC can automatically jailbreak LLMs.


### 2. Theoretical Results
Section 4 establishes a upper bound on the length of SoC attack sequence. And I believe a method which has a mathematical proof is more solid than existing jailbreak works.

weaknesses:
### 1. Poor readability
In the abstract and in the overview (Figure 1) , the authors mention "multi-armed bandit (MAB)", but they do not explain what is MAB in the introduction.I suggest the authors revise this, it confuses me until I have read related works.

In addition to MAB, the introduction has unreasonable content. With four paragraphs, more than half introduction is irrelevant to the contribution of this paper. And in the third paragraph, there are too many concepts are proposed but lack of details. I suggest that the authors adjust their introduction, current version has poor readability.

I think it is not appropriate to introduce how this paper uses MAB in related work. And why is T both rounds and sequence length (*total reward over T rounds*: line133; *attack sequence length T*:line137)? What does T represent in the subsequent content?

There are a lot of abrupt concepts introduced without much explanation in context. In line 215, I can not find any information or reference about *policy $\pi$* and *action-value Q* (including cost C in line 240). I can only speculate that this has something to do with reinforcement learning. And in Algorithm 1, what does  $E_{explore}$ stand for and what does  $E_{exploit}$ stand for? I can not find any explanation.

### 2. Lack of comparison
This paper only demonstrate SoC can jailbreak LLMs, but has no comparison with prior works. I believe that there are many excellent baseline jailbreak methods. The SoC attack has similar format with the In-Context Attack[8] which also jailbreak LLMs based on context. Besides, the authors optimize their method with reinforcement learning, but the binary reward is very similar to PAIR, which use LLM optimize jailbreak prompt[9].


### 3. Few experiments
In the entire paper, only four figures in Figure 3 prove that SoC is effective, and there is a lack of comprehensive experiments from multiple angles. For example, with different hyperparameters such as J and K, I am not sure whether this method can be generalized or only performs well under certain specific parameters.

---

[8]  Jailbreak and guard aligned language models with only few in-context demonstrations

[9] Jailbreaking Black Box Large Language Models in Twenty Queries

questions:
### 1. Related Works
I believe that in the `White-Box Attack`, the authors should not cite a lot of attack but is irrelevant to jailbreak. Besides GCG, there are many white-box attack, such as fine-tuning attack[1, 2], improved GCG[3], interpretability-based[4, 5]. Nevertheless, I am only making a suggestion. Whether the author makes modifications or not will not change my rating.

### 2. Why does the authors think modern LLMs avoid responding to malicious questions by classifier?
In Section 3.1 line 155-line157, the authors mention "In most instances, such queries fail to produce harmful responses and can be guarded using straightforward strategies, such as employing a classifier to flag harmful words and phrases". Current LLMs do not use those filter, but are fine-tuned to align with human values[6, 7].

### 3. There is no update for $\pi$ or $\pi^{*}$
Algorithm 1 aims to optimize a policy $\pi$, however, where is update for $\pi$ or $\pi^{\star}$. Why can Algorithm 1 obtain a optimized policy $\pi^{\star}$? This confuses me as to how Soc Attack works.

### typos
1. Incorrect use of quotation marks: line 153-155, 
2. Do B and C refer to the appendix? line 201 & 202 & line 318 & line 321 & line 365
3. Do you mean harmful or harmless? line 213-214: *which assigns a binary reward indicating whether the response is harmful or unsafe.*
4. *see-D.0.1*, *see-D.0.2*, *see-D.0.3* and *see-D.0.4* mean Appendix D.0.1, Appendix D.0.2, Appendix D.0.3 and Appendix D.0.4?

---


[1] FINE-TUNING ALIGNED LANGUAGE MODELS COMPROMISES SAFETY EVEN WHEN USERS DO NOT INTEND TO!

[2] SHADOW ALIGNMENT: THE EASE OF SUBVERTING SAFELY-ALIGNED LANGUAGE MODELS

[3] AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs

[4] Uncovering Safety Risks in Open-source LLMs through Concept Activation Vector

[5] How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden States

[6] Training language models to follow instructions with human feedback

[7] Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
</content>
</thread_object>

</thread>
<thread>
creation date:
1730684396387

thread_type:
official_review

<thread_object>
creator: reviewer_ugpw

<content>
summary:
This paper proposes a novel jailbreaking attack paradigm named the sequence of contexts (SoC) attacks. By leveraging techniques in multi-armed bandit (MAB), this paper maximizes the likelihood of a successful jailbreaking attack (decided by CSQ). A theoretical upper bound for the gap between the obtained and optimal rewards is presented. Experimental results show that the proposed strategy indeed enhances the effectiveness of jailbreaking attacks.

strengths:
**About novelty**

+ This paper studies jailbreaking attacks from an MAB perspective, bringing new insight into this area of research.

**About contribution**

+ This paper proposes a DMQ dataset that includes 3000 queries collected from previous works. The following works can use this as a benchmark.
+ The experimental results (figures 2 and 3) demonstrate that the proposed method enhances the effectiveness of jailbreaking methods, compared to the naive strategy. 

**About presentation**

The presentation is clear. The algorithms and figures are well-made.

weaknesses:
**About contribution**

+ This paper does not compare the proposed jailbreak attack with the existing methods.
+ The only theorem in this paper is almost trivial, and more importantly, the assumptions and limitations of this theorem are not discussed. 
It is encouraging to include theoretical analysis in LLM research. However, the results are not strong enough to serve as "one of the main contributions" (as stated in Line 253) of an ICLR paper.

In brief, the authors claim three main contributions: creating a dataset, proposing a novel jailbreaking attack strategy, and providing a theoretical analysis. However, I think contributions (ii) and (iii) are slightly overstated. That is why I gave a score of 5 for this paper. 

**About presentation**

+ This paper contains too many acronyms. I suggest adding a list of acronyms in the appendix. Besides, the authors do not provide a detailed explanation for some of the terms (e.g., sequence of context attack, and context switching queries) at their first appearance. It would make this paper more easy to follow if the authors add some explanations for the terms and reference them (e.g., see Section xxx for detailed discussions) when the term is mentioned for the first time.

+ The citation style (i.e., the green boxes) is dazzling.

questions:
See the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730711574083

thread_type:
official_review

<thread_object>
creator: reviewer_gxrd

<content>
summary:
This paper proposes a novel Sequence of Context (SoC) jailbreak attack that leverages Multi-Armed Bandit (MAB) to automatically guide context selection. The authors provide an in-depth theoretical analysis of the upper bound on the expected sequence length. Experimental results demonstrate the effectiveness of the proposed method in jailbreaking language models.

strengths:
1. The use of MAB for automated context selection in jailbreaking is novel
2. The theoretical derivation of the upper bound for the SoC attack length is well-established
3. The experimental results effectively demonstrate the method's efficacy

weaknesses:
1. Compared to other automatic jailbreak attacks (e.g., GCG, PAIR), this method requires dataset collection and policy model training, making it more resource-intensive and time-consuming
2. The proposed method is limited to pre-defined harmful query categories, and its extensibility to unseen categories is not thoroughly investigated
3. The paper lacks comparison with state-of-the-art jailbreak attacks in terms of attack success rate and computational cost

questions:
1. Could you explain the necessity of including the direct malicious query (DMQ) in the context? Given that language models with safety alignment typically reject DMQs, would it be possible to remove DMQ from the context to reduce context length?
2. How does the proposed judgment method compare with widely-used judgment systems (e.g., Llama Guard family) that are more common in jailbreak literature?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732417760733

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to all Reviewers

comment:
**Part 2/2**


- **Clarification on API Filters for Direct Malicious Queries (DMQ):**  In response to the reviewer's comment, we have added relevant citations in **Section 3.1 (Preliminaries)** to support our statement regarding the use of filters in APIs to direct malicious queries (DMQ). These citations clarify the role of filters in mitigating harmful or malicious inputs in API systems.  We believe this additional information strengthens our argument, and we invite the reviewer to refer to the updated section for a more comprehensive explanation.

- **Testing on Larger LLMs:**  
  We appreciate the reviewer's feedback regarding the evaluation of our work on only Mistral and Llama models, particularly with the focus on smaller LLMs (up to 8B). In response, we have expanded our experiments to include larger models, specifically Llama-13B and Vicuna-13B, to better demonstrate the performance of our approach. The results for these models can be found in **Appendix B**.  We believe these additional experiments strengthen the validity of our findings and invite the reviewer to refer to the updated results for a more comprehensive evaluation.  


All new changes have been highlighted in **red-font**. We believe these revisions strengthen the manuscript and address all concerns. Please let us know if further adjustments are needed. 

Given these improvements, we kindly request that the score be reevaluated with consideration for raising it above the acceptance threshold. We sincerely appreciate your time and effort in reviewing our work and look forward to your feedback on the revised draft.

We invite all reviewers to engage with our rebuttal and modifications to the draft.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732417919942

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to all Reviewers

comment:
**Part 1/2**

We would like to sincerely thank all the reviewers for their valuable comments and suggestions. We have carefully considered all feedback and have incorporated the recommended changes into the revised draft. 


- **Comparison to State-of-the-Art (SOTA) Jailbreak Attacks:**  
In response to the reviewers' request, we have conducted experiments comparing our proposed method to SOTA automatic jailbreak attacks, evaluating both the Attack Success Rate (ASR) and Cost Per Prompt.  
  - We kindly direct the reviewers to **Section 5.5** for the relevant updates.  
  - A detailed explanation of the **ASR metric** is available in **Section 5.2**, while the **cost-per-prompt metric** is described in **Section 5.5**.  

- **Analysis of the Generalization Capability:**  
In line with the reviewer's suggestion, we have conducted an analysis of the generalization capability of our proposed method to unseen DMQ categories. The results are presented in **Section 5.6** of the revised draft.  

- **Comparison of the Proposed Judgment System to State-of-the-Art (SOTA) Judgment Systems:**  
We compare our proposed judgment system with SOTA judgment systems from the Llama-Guard family in **Appendix D.5**.  

- **Analysis of the Limitations and Non-Triviality of Our Theoretical Contributions:**  
We have analyzed various scenarios where our proposed theoretical results are not tight and have illustrated these cases in **Appendix H**. Furthermore, the broader significance of our contributions to the LLM-Safety community is also discussed in **Appendix H**.  

- **Acronyms:**  
  For convenience, a table listing all acronyms used in this paper can be found in **Appendix A**.  

- **Citation Style:**  
  To improve readability, we have updated the citation color in the revised draft.  

- **Clarification of Multi-Armed Bandit (MAB) and Preliminaries:**  
  In response to the reviewer's comment, we have revised the introduction to provide a clear explanation of Multi-Armed Bandit (MAB), ensuring that the concept is properly introduced before being referenced in the abstract and **Figure 1**. We request the reviewer to refer to the revised introduction section. Additionally, we have included a detailed explanation of the MAB method in **Section 3.1**, along with clarifications of the associated notation. Furthermore, we have also illustrated all the necessary preliminaries throughout the manuscript to ensure that readers can easily follow the content without confusion.  

- **Improvement of the Introduction Structure:**  
  We have restructured the introduction to improve readability and relevance. We removed unnecessary content and streamlined the explanation to focus on the key contributions of the paper. Specifically, we have clarified the third paragraph and provided more details to avoid introducing too many concepts without adequate context.  


- **Response to White-Box Attack Citations:**  
  We appreciate the reviewer's suggestion regarding the inclusion of additional white-box attack references. In response, we have added new citations in **Section 2 (Related Work)**, covering a broader range of white-box attacks, such as fine-tuning attacks, improved GCG, and interpretability-based approaches. We believe that our citations are relevant and closely tied to the context of our work. However, if the reviewer finds any specific reference to be problematic or irrelevant, we would be happy to reconsider and make the necessary adjustments. We invite the reviewer to highlight any particular citation that may not align with the focus of the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732602898666

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Reach out to All the Reviewers

comment:
Dear Reviewers,  

Thank you for taking the time to review our submission and provide your valuable feedback. We have carefully addressed all the comments and concerns raised during the review process, as detailed in our rebuttal.  

We kindly request you to review our responses and let us know if there are any remaining clarifications or additional experiments that you believe are necessary to improve the manuscript. Our goal is to ensure that our work meets and exceeds the acceptance threshold, and we are committed to making any further improvements required to achieve this.  

Your input is invaluable in helping us refine our work, and we deeply appreciate your engagement and support. Please feel free to let us know if there is anything else we can address to strengthen the submission.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733243961115

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Message of Appreciation

comment:
Dear Reviewers, ACs and PCs,  

Thank you for your valuable feedback and insights which helped improve our work. We deeply appreciate the time, effort, and expertise you dedicated to the peer review process.  

Best regards,  
Authors #13287
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a new Sequence of Context (SoC) attacks for LLMs, and It is novel and interesting to transform the original problem to multi-armed bandit (MAB) optimization problem. After the author's rebuttal, Almost all the reviewers increase the score, and agree to accept it. But the theory appeared in this paper is not new. So I recommend this paper as a poster.