Stochastic Monkeys at Play: Random Augmentations Cheaply Break LLM Safety Alignment

Authors: Jason Vega, Junsheng Huang, Gaokai Zhang, Hangoo Kang, Minjia Zhang, Gagandeep Singh
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: LLM, Large Language Model, safety alignment, augmentations, randomness, jailbreaks, attacks

Abstract:
Safety alignment of Large Language Models (LLMs) has recently become a critical objective of model developers. In response, a growing body of work has been investigating how safety alignment can be bypassed through various jailbreaking methods, such as adversarial attacks. However, these jailbreak methods can be rather costly or involve a non-trivial amount of creativity and effort, introducing the assumption that malicious users are high-resource or sophisticated. In this paper, we study how simple random augmentations to the input prompt affect safety alignment effectiveness in state-of-the-art LLMs, such as Llama 3 and Qwen 2. We perform an in-depth evaluation of 17 different models and investigate the intersection of safety under random augmentations with multiple dimensions: augmentation type, model size, quantization, fine-tuning-based defenses, and decoding strategies (e.g., sampling temperature). We show that low-resource and unsophisticated attackers, i.e. $\textit{stochastic monkeys}$, can significantly improve their chances of bypassing alignment with just 25 random augmentations per prompt.

Decision: reject
Reviewer scores: 5, 5, 6, 3, 5

Reviews and discussion:
<thread>
creation date:
1730189861264

thread_type:
official_review

<thread_object>
creator: reviewer_unau

<content>
summary:
Facing the demand of high-resource and sophisticated design of prompts in jailbreaking attacks on LLM safety alignment, this paper studies how simple random augmentations to the input prompt bypass safety alignment. The authors perform an evaluation of 17 models and investigate random augmentations with multiple dimensions, such as augmentation type, model size, etc.

strengths:
1. The method proposed by the author is simple yet effective. Random augmentation is a simple modification to input prompts, yet such method can increase the success rate of harmful requests by up to 20~26%.
2. The evaluation across multiple dimensions is thorough. The authors investigate random augmentation in augmentation type, model size, quantization, fine-tuning-based defenses, and decoding strategies. And the authors reveal several observations, such as character-level augmentations tend to be more effective than string insertion augmentations.

weaknesses:
1. The authors claim their method to be "cheap", but no experiments support this point. The authors can add computation cost compared to other jailbreaking methods.
2. There is no comparison to other jailbreaking methods. The authors' statement that "adversarial attacks typically assume white-box...random augmentations are black-box...we do not compare..." is not reasonable. There also exist some jailbreaking methods against black-box models. The authors should compare with them. (https://arxiv.org/abs/2310.08419 ; https://arxiv.org/abs/2312.02119)
3. No widely-used commercial LLMs are evaluated. The authors should evaluate the proposed methods against widely-used commercial LLMs (e.g. ChatGPT), where bypassing the safety alignment is a more significant threat.
4. There is no discussion of mitigation methods against the proposed methods.

questions:
1. How safety judge work? Is it done by LLMs, human experts or word matching?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730312698052

thread_type:
official_review

<thread_object>
creator: reviewer_z9us

<content>
summary:
The paper investigates jailbreaking open-source LLMs via small adjustments to the prompt. The paper investigates two types of augmentations: a "string" level insertion of a new word, and a "character" level augmentation which edits, deletes, or inserts a character.
The paper evaluates using the SORRY-bench dataset, across a variety of small language models, showing that augmentation can improve ASR under a variety of sampling temepratures. The paper also evaluates the attack under different defenses and under model quantization.

strengths:
The method is simple and shows promise as a new form of jailbreak attack. The writing is clear and easy to understand, and the results are well-presented.

weaknesses:
I was not entirely convinced by the evaluation results; Although they looked promising, it would be much stronger if the paper also included more diverse results.

For example, evaluation with/without system prompts would help clarify limitations of the approach. Additional experiments with larger models, or with instruction-tuned models would also further convince me that the approach is robust and general (or more importantly, reveal something interesting about the nature of LLM robustness to perturbed prompts). 

I would also like to see more comparison to prior work; The paper evaluates comparing to (Huang et al 2023), which I felt was interesting as it demonstrates an attack surface that isn't covered by prior work (this point should be emphasized more).

To make this more complete, it would be interesting if the paper compared against the other cited work: Andriushchenko & Flammarion, 2024, Vega et al. 2023, and Zou et al. 2023.

questions:
Restating a few points from Weaknesses:

1) How does your method compare to prior work that adjusts prompts? While there is some evaluation comparing to Huang et al., it would be extremely interesting to build a more complete picture of the attack surface

2) How does your attack perform on instruction-tuned models and when including (or excluding) system prompts? (it's also unclear if system prompts were used during evaluation)

3) How susceptible are closed-source models to this attack? 

4) Have you tried string-level augmentations that insert words from a dictionary instead of random text? Would this meaningfully change anything?

5) I could not find any data with varying choices of p. While the paper mentions p=0.05 works well, it would be interesting to see how p changes the strength of the attack.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730345572676

thread_type:
official_review

<thread_object>
creator: reviewer_aovp

<content>
summary:
The paper investigates how simple random input augmentations can bypass safety alignment in state-of-the-art large language models. The authors demonstrate that “stochastic monkeys” (low-resource, unsophisticated attackers) can evade safety mechanisms in LLMs by applying 25 random augmentations to each prompt. Besides, they evaluate these augmentations across several dimensions, such as augmentation type, model size, quantization, fine-tuning-based defenses, and decoding strategies, highlighting that random augmentations enable bypassing of safety features with minimal resources. Notably, character-level augmentations consistently outperform string insertion in breaking alignment. Finally, the paper proposes a novel evaluation metric to evaluate safety and includes a human study to balance false positive and negative rates, furthering the robustness of their findings.

strengths:
1. The paper applies simple random augmentations as a threat model, contributing a fresh perspective to the understanding of LLM safety alignment vulnerability.

2. The study is thorough, evaluating the effectiveness of random augmentations across multiple dimensions, and the use of a human-validated metric strengthens the reliability of the findings.

3. By demonstrating that low-cost, unsophisticated methods can bypass LLM safety features, the paper reveals critical vulnerabilities that may shape future research in secure LLM deployment, particularly for sensitive or public-facing applications.

weaknesses:
1. While the authors discuss their method’s simplicity relative to adversarial attacks, a more detailed comparison against other established jailbreak and adversarial attack methods would better contextualize the effectiveness of random augmentations.

2. The study could benefit from an expanded discussion on the practical implications of these findings, especially in terms of what types of applications or LLM deployment scenarios are most at risk and possible mitigations.

3. The quantization experiment results in Figure 4 show variability across different models and sizes, but the authors provides vague explanations for these differences. Further exploring why certain models (e.g., Qwen 2) exhibit unique behavior under augmentations would enhance the study.

4. The writing of the conclusion is inconsistent with the abstract and conclusion, making it difficult to understand the main takeaway of this paper.

questions:
1. Can the authors provide a comparison of the success rate of random augmentations against the success rate of more sophisticated adversarial attacks in the same LLM models?

2. Among all these various dimensions( augmentation type, model size, quantization, fine-tuning-based defenses, and decoding strategies), can authors provide a ranking of these factors based on their significance towards the attack performance? In other words, the reader would like to know which factors should be the top concerns in their LLM applications. 

3. Have the authors considered examining the temporal robustness of this attack, i.e., does the attack success vary over repeated attempts or prolonged interaction with the same model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730627026586

thread_type:
official_review

<thread_object>
creator: reviewer_9lec

<content>
summary:
This paper examines the vulnerability of state-of-the-art large language models (LLMs) to random input augmentation attacks. It explores how different augmentation strategies, model configurations, and generation parameters affect attack success rates. The authors focus on two main augmentation types: Character-Level and String Insertion. Experimental results show that Character-Level augmentation yields higher attack success. Larger models demonstrate better resistance in general, though size alone doesn't ensure robustness. Furthermore, aggressive quantization tends to increase susceptibility to attacks, with variable effects across models. The paper also tests a fine-tuning-based defense, finding that it enhances resistance against longer attacks but remains vulnerable to shorter ones, with resistance varying based on attack lengths simulated during fine-tuning. A side effect noted is an increased rejection rate for benign prompts. Through extensive experimentation, the authors conclude that random input augmentation is a low-cost yet effective attack against LLM safety alignment, providing insights for future work in enhancing model robustness.

strengths:
1. The paper is well-organized and clearly written, with a coherent structure and logical progression that supports the arguments effectively.
2. The paper examines the impact of various aspects such as LLM parameter size, quantization strategies, sampling parameters, and fine-tuning on resisting random input augmentation attacks

weaknesses:
1. This paper focused on open-source chat models but lacks research on system prompts in the attack and defense settings
2. This paper did not discuss the impact of different tokenizer vocabulary sizes on this attack

questions:
1. From the attack perspective, the authors did not discuss how the system prompt was configured for the open-source models in the experimental setup. It’s possible they used the default system prompt for each model or left it empty. In fact, setting the system prompt with a defensive focus (e.g., instructing the model to only output safe content) could impact the results. From a defense perspective, the authors should also examine whether using a system prompt as a defense mechanism is effective.

2. The authors should discuss the vocabulary sizes of the different LLMs tested. Larger vocabulary models are more likely to contain "glitch tokens," which may not have been fully trained and could potentially be more vulnerable to the attack methods proposed. Would these models be more susceptible to such attacks due to this factor?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730708699799

thread_type:
official_review

<thread_object>
creator: reviewer_tz62

<content>
summary:
This paper investigates a simple but effective jailbreak attack using random perturbations. The experimental results demonstrate that this attack can significantly improve the chances of bypassing the safety alignment of 17 different models. Additionally, the paper examines how argumentation type, model size, and decoding strategy impact safety measures.

strengths:
1. This paper is well-written with clear problem formulation
2. This paper demonstrate how easy the safety alignment can be bypassed
3. The ablation analysis of the proposed argumentation attack is detailed

weaknesses:
The positioning and technical contribution of this paper seems to be unclear.
1. As a jailbreak attack paper, it lacks comparison with state-of-the-art blackbox jailbreak attacks (such as PAIR and DeepInception) in terms of attack success rate and computational cost. To strengthen the paper's contribution, the authors may need to include comprehensive comparisons with these baseline methods under the same evaluation settings.
2. As an analysis paper, despite being comprehensive, the conclusions are not new to the community. The techniques used (i.e., perturbation) are already widely employed to evaluate LLMs' safety and robustness (e.g., smoothllm). Furthermore, the impacts of model size, quantization, fine-tuning, and generation configurations on LLM safety have been well-studied previously.

questions:
1. How frequently do random augmented prompts affect the semantic meaning of the original malicious prompt? Since the output is binary, could this lead to false safe/unsafe classifications by the evaluator, potentially impacting the experimental results?
2. Have you considered combining random augmentations with existing jailbreak attacks to see if it can lead to better performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732846665237

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
Dear reviewers,

Thank you all immensely for your questions and critiques. We have prioritized the following requested items in our revised draft. **Red colored text in our revised draft indicates a new addition** (with the exception of the values in the appendix tables, where the color-coding indicates positive/negative values as was done in the original draft), with the bulk of changes being in the appendix.

1. Evaluation of closed-source model (GPT-4o) (**z9us**, **UNaU**)
    * **Results** (see Appendix D.1 for full details):
      - While GPT-4o is much safer than the open-source models, we are still able to find successful jailbreaks for it using random augmentations, with character deletion being almost two times more successful than the next best augmentation. We provide example jailbreaks in Figures 7 and 8 in the appendix.
2. Systematic investigation of the effects of safety-encouraging system prompt (**9Lec**, **z9us**)
    * **Results** (see Appendix D.2 for full details):
      - A safety-encouraging system prompt (we use the Llama 2 Chat system prompt for all models) can provide some level of protection for all models, but even so, applying random augmentations can still significantly increase the success rate across all models (e.g. by up to ~20% for Qwen 2).
3. Ablation on the augmentation strength p parameter (**z9us**)
    * **Results** (see Appendix D.3 for full details):
      - We observe the following interesting phenomenon: the success rate gains for character-level augmentations tends to increase as the augmentation strength increases, whereas the success rate gains for string insertion augmentations remain mostly unchanged. This further suggests that the safety alignment of LLMs can effectively “ignore” contiguous “noise” that does not impact the tokenization of the original prompt much.
4. Comparison with state-of-the-art black-box jailbreak attacks (PAIR and DeepInception) in terms of attack success and time cost (**Tz62**, **aoVp**, **UnAU**)
    * **Results** (see Appendix D.4 for full details):
      - **Success rate**: On paper, PAIR and DeepInception improve the success rates significantly more than random augmentations. Qualitatively however, the responses of “successful” attacks for PAIR and DeepInception are sometimes less harmful than for random augmentations. See Figure 11 in the appendix for an example. In short, this is because PAIR and DeepInception tend to produce jailbreaks that ”water down” the harmfulness of the original request in order to mitigate refusals, whereas random augmentations tend to maintain the full harmful nature of the original request. Future work can investigate more accurate assessments of these attacks that better take into account the differences in response harmfulness quality. Either way, **note that our goal with random augmentations is not necessarily to beat the state-of-the-art in black box attacks, but rather to investigate an underexplored vulnerability in LLM safety alignment.**
      - **Time cost**: Random augmentations are much faster to execute than PAIR and DeepInception, with DeepInception being more than twice as slow and PAIR being over four times as slow. Moreover, random augmentations elicit much more concise jailbroken responses than PAIR and DeepInception; on average DeepInception induces nearly twice as long target model responses as random augmentations.
5. Provided discussion on practical implications of our findings: risk assessment, possible mitigations, and a ranking of the various factors we explore in terms of how much they impact safety (**aoVp**, **UNaU**)
    * See Section 4.4 and Appendix B.
6. Greatly expanded discussion of related work to better contextualize the contributions of our work (**Tz62**)
    * See updated lines 76-82, 186-189, and Appendix A.
7. Updated the conclusion to more concisely clarify the overall takeaway of the paper (**aoVp**)
    * See Section 5.

These summarize the major reviewer items we have addressed in our paper revision. Please see our individual responses for comments on other reviewer items.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733313431018

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal Summary

comment:
Dear Area Chairs,

The major revisions to our paper are summarized under our Global Response. As shown below, we believe we have addressed each reviewer's core concerns. As of writing, 1. **Reviewer aoVp has raised their intial score of 3 to a 6**, and 2. Although we have yet to hear back from the other reviewers, **our efforts to address their concerns in addition to Reviewer aoVP's score raise provides strong reason to believe that the remaining Reviewers would find our revisions in a similarly favorable light**. We therefore believe that our revised submission **should lean towards an acceptance**.

Thank you in advance for reviewing our rebuttal; we are greatly appreciative of your time!

Sincerely,

Authors
</content>
</thread_object>

</thread>


Metareview: