BOOST: Enhanced Jailbreak of Large Language Model via Slient eos Tokens

Authors: Jiahao Yu, Haozheng Luo, Jerry Yao-Chieh Hu, Wenbo Guo, Han Liu, Xinyu Xing
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: large language model, Jailbreak

Abstract:
Along with the remarkable successes of Language language models, recent research also started to explore the security threats of LLMs, including jailbreaking attacks. Attackers carefully craft jailbreaking prompts such that a target LLM will respond to the harmful question. Existing jailbreaking attacks require either human experts or leveraging complicated algorithms to craft jailbreaking prompts. In this paper, we introduce BOOST, a simple attack that leverages only the eos tokens. We demonstrate that rather than constructing complicated jailbreaking prompts, the attacker can simply append a few eos tokens to the end of a harmful question. It will bypass the safety alignment of LLMs and lead to successful jailbreaking attacks. We further apply BOOST to four representative jailbreak methods and show that the attack success rates of these methods can be significantly enhanced by simply adding eos tokens to the prompt. To understand this simple but novel phenomenon, we conduct both theoretical and empirical analyses. Our analysis reveals that (1) adding eos tokens makes the target LLM believe the input is much less harmful, and (2) eos tokens have low attention values and do not affect LLM's understanding of the harmful questions, leading the model to actually respond to the questions. Our findings uncover how fragile an LLM is against jailbreak attacks, motivating the development of strong safety alignment approaches.large language model, Jailbreak

Decision: reject
Reviewer scores: 6, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730100950832

thread_type:
official_review

<thread_object>
creator: reviewer_q7ln

<content>
summary:
The authors investigated jailbreak attacks on large language models. The authors demonstrated that appending several EOS tokens to the end of a given sentence can significantly enhance the attack effectiveness. They explained that this improvement arises because EOS tokens bring the samples closer to the ethical boundary, as evidenced through empirical testing. The authors conducted some experiments to validate the effectiveness and generalizability of the proposed attack.

strengths:
* The idea of this paper is simple, making it easy to follow.

* The proposed method is relatively easy to replicate.

* The experiments are comprehensive.

weaknesses:
* Certain parts of the paper are redundant. Section 3.1 is overly lengthy in explaining the concept of ethical boundaries. This could be summarized in a single sentence, as the concept is essentially a classification boundary that determines whether the jailbreak is successful in the context of LLMs. I recommend that the authors revise this section, as the current content in Sections 3.1 and 3.2 may complicate the reader's understanding of the paper.

* The method proposed by the authors is easily circumvented. Model deployers can manually remove the extra EOS tokens, which diminishes the practical effectiveness of the attack. Moreover, it is unclear how to determine the optimal number of tokens to add. While the authors claim that this can be enumerated, doing so would increase attack costs.

* The core contribution of the paper seems limited to the idea of adding a few tokens to a sample.

questions:
Have the authors considered placing EOS tokens at different locations within the sentence?  Would this yield better attack results than merely appending them to the end?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730213765292

thread_type:
official_review

<thread_object>
creator: reviewer_2teg

<content>
summary:
This paper proposed BOOST (Enhanced Jailbreak of Large Language Model via Silent eos Tokens), which aims to enhance jailbreak attacks on LLMs by appending eos tokens to input prompts. The study reveals that this method can significantly improve the ASR of existing jailbreak strategies by shifting the hidden representations of harmful prompts towards harmless concept spaces, thus bypassing ethical boundaries. 

Experiments conducted on 12 different LLMs, including Llama-2, Qwen, and Gemma, demonstrated that BOOST is a general strategy that effectively enhances attack performance. Additionally, the study finds that eos tokens can be used as an effective jailbreak strategy on their own, comparable to other jailbreak methods.

strengths:
1. This paper reveals the existence of interesting ethical boundaries between benign queries and malicious queries. And found that appending <eos> token can make both benign and malicious queries closer to the boundary. The author also gave an explanation for this phenomenon: <eos> tokens are regarded as neutral during the model fine-tuning stage. The authors proposed BOOST based on this observation: simply appending eos tokens after malicious queries can effectively improve the attack success rate.

2. This paper analyzes the attention map after appending eos tokens and found that adding eos tokens wouldn't affect the original semantics of the malicious query and thus won't degrade the harmfulness of the potential model response.

3. Experimental results showed that BOOST alone can be an effective jailbreak method. And by integrating BOOST, existing jailbreak attack methods would be greatly improved.

weaknesses:
* Although the paper presents an innovative jailbreak technique, its practical effectiveness may be limited since many LLMs have built-in mechanisms to filter specific tokens, including the <eos> token, potentially rendering the attack method less viable in real-world scenarios.

* While this research explores the ethical boundaries of LLMs and demonstrates how the <eos> token can push queries closer to these boundaries, it overlooks a crucial aspect: users typically customize system prompts based on their specific needs, which inherently shifts these ethical boundaries. Therefore, a more comprehensive analysis of how the <eos> token's effectiveness varies across different system prompts would strengthen the study's findings.

* The author only discussed the character-perturbation type jailbreak defenses, which is not very practical as these methods would affect the utility of the model's performance on nominal queries.

questions:
**Question 1** The experimental results show that the eos token pulls malicious queries toward ethical boundaries, which reminds me of how data points near decision boundaries typically exhibit higher uncertainty in classification problems. Could the author test BOOST's effectiveness against Gradient Cuff [1], which detects jailbreak prompts by analyzing the gradient norm (the gradient nom is to some extent the uncertainty based on my understanding)

**Question 2** As I mentioned in weakness 2, could the author test the BOOST's performance against system-prompt-engineering defenses like Self-Reminder [2]? It could be more supportive if the author also visualized the t-sne plot of LLMs when system prompts changed. 

****
**References**

[1] Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes. Xiaomeng Hu, Pin-Yu Chen, Tsung-Yi Ho.

[2] Defending ChatGPT against jailbreak attack via self-reminders. Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, Fangzhao Wu.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730538631501

thread_type:
official_review

<thread_object>
creator: reviewer_iafp

<content>
summary:
The paper introduces BOOST, a new method for improving jailbreak attacks on LLMs. Unlike previous methods that require complex prompt engineering, BOOST simply appends EOS tokens to harmful prompts, leading to the successful bypassing of LLMs' ethical filters. The authors show that EOS tokens can shift harmful prompt representations toward benign concepts, making it easier to evade ethical boundaries in LLMs. The paper demonstrates BOOST's effectiveness across multiple LLMs and jailbreak methods, significantly enhancing the attack

strengths:
1.	BOOST’s simplicity and effectiveness jailbreak attack methods introducing EOS token manipulation as a jailbreak approach.
2.	The experiments are thorough, using a range of LLMs and showing detailed quantitative results.
3.	The methodology and analysis are clearly presented, with visualizations of ethical boundaries and attention shifts, aiding in understanding the impact of EOS tokens.

weaknesses:
1.	BOOST’s effectiveness is limited for proprietary models, which may filter EOS tokens, potentially reducing BOOST’s applicability.
2.	The mechanism by which EOS tokens affect model behavior may vary, necessitating further exploration of token influence across different model architectures.
3.	BOOST's efficiency relies on finding the optimal number of EOS tokens, which might introduce variability and inconsistency in attack performance.

questions:
1.	Could other special tokens (e.g., BOS) offer similar effects as EOS in bypassing ethical boundaries?
2.	How does BOOST compare in effectiveness and reliability with proprietary filtering mechanisms?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706567884

thread_type:
official_review

<thread_object>
creator: reviewer_qcrm

<content>
summary:
This paper proposes a simple method to bypass the safety training of LLMs by appending EOS tokens to the user prompt. This can be used in combination with other attacks such as GCG. The authors show that their attack can significantly improve the attack success rate on a sample of AdvBench prompts. They further provide some justification for the success of their attack by 1. Demonstrating that a learned “ethical boundary” exists, 2. Analyzing how EOS tokens can push representations across the ethical boundary, and 3. Showing how attention values are low for EOS tokens, avoiding the problem of “empty jailbreaks” [1].

[1] Souly, A., Lu, Q., Bowen, D., Trinh, T., Hsieh, E., Pandey, S., Abbeel, P., Svegliato, J., Emmons, S., Watkins, O. and Toyer, S., 2024. A strongreject for empty jailbreaks. arXiv preprint arXiv:2402.10260.

strengths:
1. The proposed jailbreaking technique is simple to understand and apply.
2. The paper provides empirical evidence to support the idea that LLMs learn an “ethical boundary” during alignment, and that various attacks (including the proposed EOS attack) can push representations across this boundary.
3. The evaluation results appear rigorous. Human evaluation was performed to estimate the accuracy of their evaluation metrics, demonstrating reasonably high accuracy (92%). Where applicable, reported results were averaged over multiple trials with means and standard deviations reported.

weaknesses:
1. The threat model assumes the attacker is able to add EOS tokens to the user prompt. To the best of my knowledge, no popular closed-source model (e.g. ChatGPT, Claude) allows this. In practice, this jailbreak is therefore mostly just applicable in the open-source setting, as addressed in the limitations section.
2. I’m not very convinced by the proposed explanation of why adding EOS tokens can successfully jailbreak the models (lines 267-288). The explanation is that EOS tokens are considered “neutral” since they always appear at the end of (prompt, response) pairs during RLHF. But its still unclear why the act of adding neutral tokens would proactively induce a shifting behavior, as opposed to having no effect on the ethical classification.

Minor suggested changes: Line 11: “Large language models” -> “Large Language Models (LLMs)”; Line 127, 136: “tokes” -> “tokens”; Line 201: “as well the” -> “as well for the”; Figure 3: add legend for harmful vs. benign colors; Lines 299-300: “still affects the response” is oddly phrased, perhaps change to “the response can be disproportionately affected by the attack itself”; Line 311: “empty jailbreak” -> “an empty jailbreak”; Line 702: “Disclose” -> “Disclosure”

questions:
1. Can you clarify the notation in lines 152-154? In the first and third conditional probabilities, the random variable x is replaced by a condition “if x is (un)ethical.” I find this notation to be strange and unclear. At a high level, what is trying to be communicated here? Is it that, depending on z, the responses become entirely concentrated on $\mathcal{R}_\text{refuse}$ or its complement?
2. Just to clarify: for the t-SNE visualizations, are the plotted hidden representations 1. A concatenation of the representations across all tokens, 2. The representation of just the final token of x, or 3. Something else?
3. Can you provide some explanation for why you use different visualization methods for figures 2/3 and 4 (t-SNE vs. PCA)?
4. Figure 4 shows that both harmful and benign prompts are shifted towards the boundary. On one hand, this means that the model is being encouraged to comply with harmful prompts. But wouldn’t this mean that on the other hand, the model is also being encouraged to refuse benign prompts? Have you observed such refusals of benign prompts from adding EOS tokens? Does this occur as often as jailbreak success of harmful prompts?
5. In figures 5 and 12, what does the horizontal axis represent? Consider labeling this in the figure.
6. Lines 446-448 explains that the sensitivity of ICA/CO+EOS attack success to the number of EOS tokens added is due to the fact that EOS tokens can push the representations over the boundary in either direction. However, you could also say the same thing might happen for GCG/GPTFuzzer. So why are ICA/CO different?
7. In figure 8, could the reason why other tokens aren’t as successful as EOS be that the attention values are much higher? Can there be some experiments added to check this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731971298790

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revision & General Response Rebuttal

comment:
Dear Reviewers,

  

We thank the reviewers for the insightful questions and reviews. Your time and effort dedicated to improving our work are truly appreciated.

  

We have done all the experiments suggested and answered all the questions. All modifications are marked in red color.

  

Major revisions include:

  

-   **New closed-sourced model experiment result in Appendix E.9**. `all reviewers`
    

	-   Conducted an ablation study to probe four closed-source models (Claude-3-opus, GPT-4o, Gemini-1.5-pro, Qwen-max) for their handling of eos tokens.
    
	-   Results: Discovered that GPT-4o and Qwen-max had released tokenizers while probing revealed that 3 out of 4 models do not filter eos tokens, exposing vulnerabilities to injection attacks. Additionally, testing BOOST on GPT-4o-mini and Qwen-max with GPTFuzzer showed that BOOST enhances the ASR, emphasizing the importance of robust input filtering mechanisms
    

-   **New limited iterations number experiment in Appendix E.8.** `all reviewers`
    

	-   Conducted an ablation study to evaluate the sensitivity of non-optimization-based methods to the number of appended eos tokens, limiting the search budget to only 5.
    
	-   Results: While a slight drop in ASR was observed, BOOST still achieved a higher ASR than the baseline with a query budget of 5. This demonstrates that BOOST is efficient in improving non-optimization-based methods.
    

-   **New eos position experiment in Appendix E.11** `reviewer q7Ln`
    

	-   Conducted an ablation study to analyze the effectiveness of placing eos tokens at different locations (beginning, middle, and random positions) within prompts for Llama-2-7B-chat using GPTFuzzer
    
	-   Results: BOOST achieved the highest performance when eos tokens were appended at the end of prompts while placing them in other positions resulted in lower success rates. This confirms that appending eos tokens at the end is the most effective strategy for bypassing model safeguards.
    

-   **New analysis eos influence on the benign prompt experiment in Appendix C.4.** `reviewer QCrM`
    

	-   Conducted an ablation study to assess the impact of appending 5 eos tokens to 256 curated benign prompts (§C.1).
    
	-   Results: Found that 41 benign prompts were refused after appending eos tokens. For instance, as shown in Figure 12, the model responded appropriately without eos tokens but refused the same benign prompt when eos tokens were added. This supports our hypothesis that appending eos tokens shifts benign prompts toward the ethical boundary, consistent with the findings in §3.3.
    

  

-   **New other token shift experiments in Appendix C.5.** `reviewer iaFP`
    

	-   Conducted an ablation study to visualize the hidden representations of prompts appended with various tokens, as shown in Figure 13.
    
	-   Results: Appending eos tokens shifted hidden representations closer to benign prompts, effectively bypassing the ethical boundary. The bos token had the second smallest distance to the benign prompt center, followed by the unk token, while other tokens showed the minimal impact. These findings align with the analysis in §6, highlighting the unique effectiveness of eos tokens in altering model behavior.
    

-  **New other token Attention value experiments in Appendix E.10.** `reviewer QCrM`
    

	-   Conducted an ablation study to visualize the attention values of various tokens for the Gemma-2B-it model in the -10th layer and 0th head.
    
	-   Results: The attention values of other tokens were significantly higher than those of eos tokens, suggesting that these tokens are more likely to distract the model's attention from the original content. This highlights why eos tokens are particularly effective in bypassing the ethical boundary without altering the model’s focus on the prompt.
    

-   **New defense methods experiments in Appendix E.6.** `reviewer 2Teg`
    

	-   Conducted an ablation study to evaluate the robustness of BOOST against defense methods: RPO, SmoothLLM, Self-Reminder, and Gradient Cuff (GC), on four models (Gemma-2B-Instruct, LLaMA-2-7B-Chat, LLaMA-2-13B-Chat, LLaMA-3-8B-Instruct).
    
	-   Results: BOOST can slightly enhance attacks. These defenses, especially GC, significantly reduce the ASR. Additionally, visualizations (Figure 18) reveal that the Self-Reminder defense enhances the separation between benign and harmful prompts, effectively strengthening the model’s ethical boundary and making jailbreak attacks more difficult.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731971713849

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response Q1

comment:
> **Reviewer's Comment:** Close-sourced model would filter eos tokens `all reviewers`

**Response:** Thank you for raising this important concern. We acknowledge that closed-source models may implement filtering mechanisms to exclude eos tokens, which could potentially limit the applicability of our method. **We had noted this as a potential weakness in Section 6 of our initial submission.**

However, we would like to emphasize that our assumption about the accessibility of eos tokens is significantly more practical than assuming access to model gradients, as required by white-box attacks. To address your concern further, we conducted an additional study on four leading closed-source LLM providers: OpenAI, Anthropic, Qwen, and Gemini—whose models are leading on the LLM arena leaderboard [1].

We investigated whether eos tokens are filtered in their APIs by probing the models with specific queries. Specifically, we used the prompt: _“Can you repeat {eos} ?”_ where `{eos}` is either the officially released eos token (if available) or common eos tokens. We use _“what’s this token in LLM: {eos} ?”_ alternatively if the model is trained to refuse to repeat due to prompt injection concern. The probing process works as follows:

- **Successful Probing (Not Filtered):** If the model's eos token is not filtered and matches the one we used, the model responds with something like "Sure, here it is" followed by an empty or truncated output. This truncation occurs because the eos token signals the end of generation, causing the model to stop generating further text. This indicates that the model recognizes and does not filter the eos token.

- **Filtered eos Token:** If the model responds with "Sure, please provide me the word you want me to repeat," it suggests that the eos token is filtered out—meaning the model did not process it due to input filtering mechanisms. This implies that while the token might be recognized internally, it is being excluded from the input the model processes.

- **Unsuccessful Probing:** If the model provides a different response, it indicates that the used token is not the model's actual eos token.

**Probing Results:**

Our findings are in Figure 22 and summarized below:

- **GPT-4o and Qwen-max:** Both models have officially released their tokenizers. We successfully probed their eos tokens and found that they do not filter out eos tokens in their API services.

- **Claude-3-opus:** Although the tokenizer is not officially released, we successfully probed its eos token using common eos tokens and confirmed that it does not filter out eos tokens.

- **Gemini-1.5-pro:** We were unable to successfully probe this model's eos token, possibly due to a lack of information about its tokenizer.

**Implications:**

These results indicate that **three out of the four models can be successfully probed and all of three do not filter out eos tokens** in their API services. This suggests that our method can applicable to a significant portion of popular closed-source models. We have added the details and screenshots in Appendix E.9.

Furthermore, we applied our BOOST method in conjunction with GPTFuzz on two of these models, GPT-4omini and Qwen-max. The results demonstrate that BOOST effectively enhances attack success rates even against these closed-source LLMs. This indicates that, contrary to the assumption that closed-source models would filter eos tokens, our method remains effective in practice.

For ethical considerations, we have disclosed these findings to the relevant providers to highlight the importance of user input filtering. We believe that the ultimate goal of jailbreak research is to improve the safety and robustness of LLMs. Our unique findings and subsequent disclosures serve as an important alert for LLM providers to carefully examine and strengthen their token-filtering mechanisms.

Therefore, our work retains significant value for closed-source LLMs and provides actionable insights for their developers. By demonstrating that eos tokens are not universally filtered and that our method can enhance attack success rates even on closed-source models, we contribute valuable knowledge that can help improve the security practices of LLM providers.

[1] Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
</content>
</thread_object>

</thread>
<thread>
creation date:
1733265048141

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal and Discussion Summary

comment:
**Dear Area Chair,**

We would like to express our sincere gratitude to all the reviewers for their insightful comments and recognition of our work. We are encouraged that the reviewers highlighted the following strengths in our paper:

-   **Simplicity and Effectiveness of BOOST**: Reviewers QCrM, iaFP, 2Teg, and q7Ln appreciated that our proposed method is simple to understand, easy to replicate, and effectively enhances jailbreak attacks on LLMs.
    
-   **Comprehensive Evaluation**: Reviewers QCrM, iaFP, 2Teg, and q7Ln acknowledged that our experiments are extensive and rigorous, involving a wide range of models and providing detailed quantitative results.
    
-   **Clear Presentation and Analysis**: Reviewers QCrM, iaFP and 2Teg noted that our methodology and analysis are clearly presented, with visualizations aiding in understanding the impact of EOS tokens.
    
-   **Novel Insights into Ethical Boundaries in LLMs**: Reviewers QCrM and 2Teg recognized our empirical evidence supporting the concept of an "ethical boundary" learned during model alignment and how EOS tokens can influence this boundary.
    
---
Below, we list the main concerns raised by the reviewers and highlight how we have addressed them during the discussion period.

1.  **Closed-Source Models**  
    _(Raised by all reviewers)_
    
By EOS probing, we discovered that several popular closed-source models, including **GPT-4o**, **Claude-3**, and **Qwen-max**, do not filter out EOS tokens in their API services. We successfully applied BOOST to two of these models, demonstrating its effectiveness in real-world scenarios. 
        
2.  **Number of EOS Tokens**  
    _(Raised by Reviewers QCrM, iaFP, q7Ln)_
    
 We clarified that for optimization-based methods like GCG and GPTFuzzer, we do not need to change the number of EOS tokens as they can adjust iteratively. Then, we conducted an ablation study with a reduced maximum query budget of just **5** iterations on three models. Even with this limited number, BOOST still significantly enhances ASR.
    
**Reviewer-Specific Responses:**

**Reviewer QCrM:**

   -   We expanded our explanation, suggesting that EOS tokens impact context segmentation due to their consistent use during RLHF training, effectively diluting the influence of disallowed content.
    -   We conducted experiments showing that appending EOS tokens can cause models to refuse benign prompts, supporting our hypothesis about shifting towards the ethical boundary.
    -   We conducted additional experiments comparing attention values of different tokens.

We address Reviewer QCrM's concerns and further clarified how we probe the EOS tokens.
    

----------

**Reviewer iaFP:**
    
   -   We investigated other tokens like BOS and found they are not as effective as EOS tokens in shifting representations to bypass ethical boundaries .
    -   We acknowledged that the effect of EOS tokens may vary across model architectures and suggested this as an area for future exploration.

The reviewer did not provide further responses. However, similar concerns were acknowledged and addressed by other reviewers, and we believe we have thoroughly addressed their concerns.
    

----------

**Reviewer 2Teg:**

   -   We conducted experiments testing BOOST against GC and Self-Reminder.  
    -   We included t-SNE visualizations showing how defenses like Self-Reminder enhance the separation between benign and harmful prompts, reinforcing the ethical boundary.

The reviewer acknowledged that their concerns were addressed and raised the score.
    

----------

**Reviewer q7Ln:**
    
   -   We revised Section 3.1 to be more concise.
    -   We clarified that our contributions extend beyond adding tokens, including systematic study of EOS token influence, comprehensive validation across diverse models and methods, and novel insights into LLM vulnerabilities.
    -   We conducted experiments placing EOS tokens at different positions and found that appending them at the end yields the best performance, aligning with model training behavior.
    -  We further clarified the novelty, evidence and insights of our approach.

The reviewer acknowledged that the concerns were addressed and acknowledged our further clarification.
    
---

We believe we have addressed all concerns raised by the reviewers thoroughly and we have added **7 pages** for additional experiments and discussions suggested by reviewers. Our work offers a new insight to understanding LLM vulnerabilities, presenting a simple yet effective method for jailbreak. We have  demonstrated the practical effectiveness of BOOST through comprehensive experiments and analyses.

We kindly ask for your consideration of our paper for acceptance. We are confident that our work will be a valuable addition to the conference and contribute to the field of AI security.

**Thank you for your time and consideration.**

Sincerely,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposed an attack by adding <eos> tokens at the end of the prompts can realize jailbreak. The authors also proposed a hypothesis to explain such an attack and the viewpoint is that the <eos> token helps question to move across the ethical boundary. After rebuttal, there are two negative scores and two positive scores. The main issues are whether this attack can be implemented in commercial LLMs and whether the explanation is reasonable. After reading all comments and rebuttals, I think these two issues are not well addressed and the paper is not ready to publish.