h4rm3l: A Language for Composable Jailbreak Attack Synthesis

Authors: Moussa Koulako Bala Doumbouya, Ananjan Nandi, Gabriel Poesia, Davide Ghilardi, Anna Goldie, Federico Bianchi, Dan Jurafsky, Christopher D Manning
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: LLM safety, program synthesis, compositional modeling, jailbreak attacks, red-teaming, domain-specific languages, string transformations, AI safety research, black-box optimization, automated benchmarking

Abstract:
Despite their demonstrated valuable capabilities, state-of-the-art (SOTA) widely deployed large language models (LLMs) still have the potential to cause harm to society due to the ineffectiveness of their safety filters, which can be bypassed by prompt transformations called jailbreak attacks. Current approaches to LLM safety assessment, which employ datasets of templated prompts and benchmarking pipelines, fail to cover sufficiently large and diverse sets of jailbreak attacks, leading to the widespread deployment of unsafe LLMs. Recent research showed that novel jailbreak attacks could be derived by composition; however, a formal composable representation for jailbreak attacks, which, among other benefits, could enable the exploration of a large compositional space of jailbreak attacks through program synthesis methods, has not been previously proposed. We introduce h4rm3l, a novel approach that addresses this gap with a human-readable domain-specific language (DSL). Our framework comprises: (1) The h4rm3l DSL, which formally expresses jailbreak attacks as compositions of parameterized string transformation primitives. (2) A synthesizer with bandit algorithms that efficiently generates jailbreak attacks optimized for a target black box LLM. (3) The h4rm3l red-teaming software toolkit that employs the previous two components and an automated harmful LLM behavior classifier that is strongly aligned with human judgment. We demonstrate h4rm3l's efficacy by synthesizing a dataset of 2656 successful novel jailbreak attacks targeting 6 SOTA open-source and proprietary LLMs (GPT-3.5, GPT-4o, Claude-3-Sonnet, Claude-3-Haiku, Llama-3-8B, and Llama-3-70B), and by benchmarking those models against a subset of these synthesized attacks. Our results show that h4rm3l's synthesized attacks are diverse and more successful than existing jailbreak attacks in literature, with success rates exceeding 90% on SOTA LLMs. Warning: This paper and related research artifacts contain offensive and potentially disturbing prompts and model-generated content.

Decision: accept (poster)
Reviewer scores: 6, 8, 5, 8

Reviews and discussion:
<thread>
creation date:
1729451011342

thread_type:
official_review

<thread_object>
creator: reviewer_onbh

<content>
summary:
The paper proposes h4rm3l, a domain-specific language designed to describe and facilitate jailbreak attacks on language models. h4rm3l addresses the gap between the diversity of jailbreak attack techniques and the limited benchmarks currently available. It formally expresses jailbreak attacks as compositions of parameterized string transformation primitives, enabling it to capture a wide range of attack types and their combinations. Furthermore, h4rm3l incorporates a bandit-based approach to jailbreak attacks and synthesizes a dataset containing 2,656 successful novel attacks targeting six state-of-the-art (SOTA) open-source and proprietary LLMs. The results demonstrate that the synthesized attacks are diverse and significantly more successful than previously reported methods, achieving success rates exceeding 90% on these SOTA models.

strengths:
h4rm3l offers a formalized framework for representing jailbreak attacks, addressing the need for a consistent, standardized approach in this area of research. By utilizing parameterized string transformation primitives, it enables the modeling of diverse and complex attack types, making it a versatile tool. Notably, h4rm3l has demonstrated impressive empirical success, achieving over 90% success rates in generating jailbreak attacks across several state-of-the-art language models, including GPT-4o, Claude-3, and Llama3. This showcases its robustness and effectiveness in exposing vulnerabilities in a wide range of LLM architectures, proving its scalability and relevance for both proprietary and open-source systems.

weaknesses:
While the paper is impressive, it does not sufficiently address its limitations. Beyond the template-based jailbreak methods and synthesis, other forms of jailbreak techniques exist, such as fuzzing or mutation-based or other approaches ([1], [2], [3]). Given that the authors propose a language to express various attack strategies, it is important to clarify the scope of h4rm3l in this regard. The paper should discuss how many types of jailbreak methods the language can effectively cover and whether it faces any limitations in representing certain attacks. Furthermore, it would be beneficial if the authors included a more comprehensive discussion of related work in the background section, addressing whether h4rm3l can accommodate or has limitations in expressing these alternative attack types.

[1] https://www.usenix.org/conference/usenixsecurity24/presentation/yu-jiahao
[2] https://www.usenix.org/conference/usenixsecurity24/presentation/yu-zhiyuan
[3] https://www.usenix.org/conference/usenixsecurity24/presentation/liu-tong

questions:
1. To what extent can h4rm3l cover alternative jailbreak techniques, such as fuzzing or mutation-based attacks, beyond template-based methods?
2. Could the paper further discuss how h4rm3l compares to other jailbreak methodologies and clarify whether it can fully accommodate these approaches or has constraints?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730469045836

thread_type:
official_review

<thread_object>
creator: reviewer_rpnp

<content>
summary:
This paper introduces h4rm3l, a novel approach to cover sufficiently large and diverse sets of jailbreak attacks with human-readable domain-specific language. h4rm3l includes (1) a domain-specific language formally expresses jailbreak attacks as compositions of parameterized string transformation primitives, (2) a synthesizer with bandit, and (3) a red-teaming software toolkit. h4rm3l also provides a dataset of 15891 novel jailbreak attacks and safety benchmarking results for 6 SOTA LLMs with 82 jailbreak attacks.

strengths:
1. This paper studies jailbreak attacks from a novel perspective like software engineering, to synthesize (all) jailbreak attacks and evaluate the jailbreak robustness of LLMs.

2. The experiment is comprehensive and the visual analysis is clear.

3. The discussion part is detailed.

weaknesses:
1. Section 3.1, especially line 166-167, makes this reviewer confused. It seems that Section 3.2 has few connections with Section 3.1. To specific, the description in Section 3.1 does not significantly contribute to this reviewer's understanding of the connection from the motivation to the method of this paper. For example, this reviewer wants to know more details about the generic decorator TransformFxDecorator and why it covers the space of all string-to-string transformations and could represent any jailbreak attack on black-box LLMs.

2. Based on the first point, the reviewer is concerned that the author's claim of “representing all black box attacks” is an overclaim. As the author cited in Section 2, PAIR, and its following version TAP, employ a red-teaming assistant to jailbreak the target model. So the algorithm output is uncertain. The authors should explain the inclusive relationship between h4rm3l and other black-box jailbreak attacks.

3. According to this reviewer's understanding, the process of synthesizing jailbreak attacks briefly includes two sub-processes: selection, composition, and evaluation. Section 3.2 and 3,3 describe the details of selection and evaluation, respectively. However, how are the selected jailbreak methods combined in this paper?

questions:
This reviewer's questions focus on the methods' details. See weakness 1-3. This reviewer wants to know more about the details of this method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730649262171

thread_type:
official_review

<thread_object>
creator: reviewer_fdvc

<content>
summary:
This paper introduces h4rm3l, a domain-specific language (DSL) for representing jailbreak attacks on large language models (LLMs) as compositions of parameterized string transformation primitives. The authors develop a synthesizer using bandit-based few-shot program synthesis to generate novel, high-success-rate jailbreak attacks against black-box LLMs. They demonstrate the effectiveness of h4rm3l by synthesizing 2,656 successful attacks targeting six state-of-the-art LLMs, achieving high success rates. The work also includes a zero-shot harmful LLM behavior classifier and an open-source red-teaming toolkit.

strengths:
- The paper addresses a timely and relevant topic.
- The structure of the writing is clear and well-organized.
- The paper presents an innovative idea by transforming the jailbreak task into a formal language implementation.
- Implementing various existing jailbreak methods to support the proposed composite attack strategy.

weaknesses:
- Unclear Motivation and Purpose: The motivation and unique advantage of transforming jailbreak tasks into a formal language are not entirely clear. In my view, directly developing a framework or tool that integrates multiple jailbreak prompting operators and scheduling strategies may already suffice for most jailbreak/red-teaming needs. Therefore, what specific advantages or unique capabilities does this formal language offer? How does it surpass the functionalities of traditional red-teaming frameworks? Additional clarification from the authors would be helpful.

- Assessment Method in Section 3.3: Using GPT as a judge has become a mainstream approach for evaluating jailbreak success rates, and many recent studies follow standard methods. However, this paper appears to introduce a new prompt. Is this prompt demonstrably superior to those used in other jailbreak studies? Alternatively, what necessary adaptations or unique design choices were made to tailor this prompt for the work presented here?

- Efficiency Comparison in Experiments: While the authors clearly present the effectiveness of the proposed method compared to baselines in jailbreak success rates (as shown in Fig. 4), a comparative analysis of jailbreak efficiency with existing methods is lacking. Figures 2 and 3 only show the number of iterations required by the proposed method. I suggest the authors provide a more straightforward and detailed efficiency comparison with baseline methods.

questions:
1. Compared to directly building a red-teaming framework, what are the unique advantages of the formal language proposed in this paper?
2. Does the proposed jailbreak attack demonstrate superior efficiency compared to the baselines discussed in the paper?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730651063866

thread_type:
official_review

<thread_object>
creator: reviewer_cwn6

<content>
summary:
The paper introduces h4rm3l, a domain-specific language (DSL) designed to formally represent jailbreak attacks on large language models (LLMs) as compositions of parameterized string transformations. The authors propose a framework that includes the h4rm3l DSL, a synthesizer with bandit algorithms for generating optimized jailbreak attacks, and a red-teaming software toolkit. The paper demonstrates the efficacy of h4rm3l by synthesizing a dataset of successful jailbreak attacks targeting six SOTA LLMs, showing higher success rates than previously reported attacks.

strengths:
- This paper introduces the first formal, composable representation of jailbreak attack, providing a more systematic and comprehensive approach to assessing LLM vulnerabilities.
- The efficacy of the proposed framework is effectively demonstrated through the synthesis of a substantial dataset comprising successful jailbreak attacks against multiple state-of-the-art (SOTA) LLMs.

weaknesses:
- The estimation of attack success rates relies solely on the assessment of 100 Claude-3-haiku responses by just two human annotators, which raises concerns about the generalizability of the findings.

- The relationship between various jailbreak methods and their impact on the results is not clearly articulated, particularly how the synthesis with h4rm3l connects to these outcomes.

- Some parts of the paper are vaguely written and are subject to elaboration and clarification.

- The following issues are subject to further clarification:
  - What do the numbers (e.g. "00536") in Figure 4 represent?
  - Are the horizontal and vertical coordinates of Figure 5 meaningful?
  - In line 928, "therefore we use proxy names" Do the proxy names here refer to the letters G, B and U?
  - What do the abbreviations BS, RP, and RT in Tables 3 and 4 signify?
  - In line 1222, "Table 2 and Table 3" should be corrected to "Table 3 and Table 4."?

questions:
- What are the time and computational costs associated with using h4rm3l for synthesizing attacks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732310932776

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Generality of h4rm3l and Value of its formal expression of jailbreak attacks; DRA case study

comment:
All the reviewers ask some question about the generality and value of translating jailbreak attacks into a formal language. We address this issue here, using as an illustration the DRA method [(Liu et al. (2024)](https://www.usenix.org/conference/usenixsecurity24/presentation/liu-tong) mentioned by reviewer oNbH.


$\textbf{Generality and extensibility of h4rm3l}$

- h4rm3l primitives are string-to-string functions written in python that can perform arbitrary computation on strings, and therefore can represent any black-box attack.

- New h4rm3l primitives can be added by implementing python classes inheriting from `PromptDecorator`. We implemented several primitives from the literature 
(See `h4rm3l/src/h4rm3l/decorators.py` and `h4rm3l/src/h4rm3l/decorator\_expressions.py` in supplementary materials), and Appendix B in our main manuscript.

- h4rm3l also includes two generic primitives for convenience, which researchers and practitioners can use to rapidly define custom primitives directly in h4rm3l:
`RolePlayingDecorator`, which allows defining templated attacks with a custom prefix or suffix, and `TransformFxDecorator`, which allows transforming prompts with arbitrary python code. Note that any black box attack can be represented as a single `TransformFxDecorator` instance.

- Another convenient feature of `TransformFxDecorator` is that the python function it specifies is supplied with the `assistant` parameter, which is a handle to a configurable auxiliary LLM that can be used to perform transformations such as rephrasing or translating a prompt as performed in primitives such as `TranslateDecorator` (Yong et al. (2023)).

$\textbf{Implementation of DRA in h4rm3l}$

**Listing 2** in the next post shows how to implement DRA as a single instance of `TransformFxDecorator`. Although this implementation fully expresses DRA, it doesn't fully leverage the power of h4rm3l, which lies not only in the expression of arbitrary computation on strings, but also in the compact expression of jailbreak attacks as composition of interpretable human-defined abstractions.

A better approach is to implement DRA as a composition of granular primitives (defined in next post **Listing 1**). Then the program synthesizer can usefully explore both novel parameterizations and compositions of these primitives and other defined primitives. The simplest breakdown of DRA would compose just two parameterized primitives `DisguiseDecorator(toxicRatio, benignRatio).then(ReconstructDecorator())`. However, a better solution would be to split DRA into 4 primitives, as follows.

\begin{align*}
    &WordPuzzleObfuscationDecorator()\cr
    .then(&WordCharSplitDecorator(toxicRatio, benignRatio)) \cr
    .then(&PayloadReconstructionDecorator()) \cr
    .then(&ContextManipulationDecorator())
\end{align*}

If h4rm3l's synthesizer is run with this latter set of primitives, together with other primitives we explored such as `Base64Decorator` and `RefusalSuppressionDecorator` then h4rm3l may generate novel attacks that build on DRA's components and apply additional transformations such as inserting statements instructing the target LLM to not refuse requests, applying base64 encoding on the DRA attack payload, and intructing the target LLM to start its response with an affirmative prefix. The power of our formal DSL is to allow automatic exploration of such novel compositions in the resulting large compositional space, and these novel compositions are the basis of our high attack success rates, as well as our large-scale generation of jailbreak attacks.

\begin{align*}
    &WordCharSplit(tr=0.3, benignRatio=0.1)\cr
    .then(&RefusalSuppressionDecorator()) \cr
    .then(&WordLevelPuzzleObfuscation()) \cr
    .then(&PayloadReconstruction()) \cr
    .then(&Base64Decorator()) \cr
    .then(&AffirmativePrefixInjectionDecorator())
\end{align*}

See `benchmark.html` in our supplementary materials to explore the synthesized attacks we benchmarked (click on individual programs to view their source code in h4rm3l).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732895579911

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
h4rm3l paper revision: Improved clarity and implementation details

comment:
We have updated our manuscript based on the reviewers helpful constructive feedback. See updated PDF, which highlights changes from our initial submission.

- **Abstract \& Intro**
    - Noted our finding on the necessity for target-specific attack synthesis.
    - Clarification edits.
    
- **Related Work**
    - Incorporated feedback from **Reviewer oNbH**. Clarified h4rm3l's significance in comparison with traditional redteaming frameworks as asked by **Reviewer fDVc**.
    - Referenced Wei et al. (2023)'s definition of black-box attack as transformations that induce *restricted* behaviors in LLMs.
    - Added DRA (Liu et al., 2024) as example.
    - Contrasted h4rm3l with LLM-Fuzzer (Yu et al., 2024a).
    - Explained how h4rm3l can be used in experiments such as Qiu et al. (2023)'s, which used structured templates. Distinguished *jailbreak prompts* and *jailbreak attacks*.
    - Contrasted h4rm3l's approach to learning from human jailbreakers with prior approaches such as (Yu et al., 2024b)'s.

- **Method**
    - Improved this section based on feedback from **Reviewers oNbH, fDVc, RpnP and CWn6**. Clarified the h4rm3l language, its composition and extension mechanisms, its relationship with Python, the reason why it covers all black-box attacks, the role of its generic primitives, and the reason why it is a suitable basis for representing, analyzing and generating jailbreak attacks.
    - Introduced Sections 3.1, 3.2 and 3.3 more coherently. The previous organization led to confusion, as **Reviewer RpnP** noted. Also clarified the groundings of our ASR estimation method in response to **Reviewer CWn6**'s notes. **Reviewer oNbH**'s questions on related works also helped present this section more coherently, and clarify how our contributions relate to prior work.
    - Replaced prior Equation. 1 (now in appendix M) with Listing 1, which gives a concrete example of synthesized jailbreak attack composing generic primitives.
    - Referenced Anthropic's Acceptable Use Policy to contextualize example in Figure 1.
    - In Equation 1 (ASR), clarified that H is also a function of the original illicit request.
    - Updated hyperparameter names ($N_{iters}$, $k_{illicit}$, $k_{examples}$) for clarity.
    - Clarified proposals generation in Algorithm 1. Made  $N_{proposals}$ hyperparameter explicit.
    - Added complexity expression to Algorithm 1. This and the wallclock time performance we added (Appendix L), and our improved discussions on the synthesizer hyperparameters address **Reviewer fDVc**'s questions on the efficiency of our synthesizer.

- **Results**
    - Improved comparison of targeted attack synthesis methods.
    - Improved captions for Figures 2, 3 and 4.
    - Based on **Reviewer CWn6**'s feedback, 
    clarified that the t-SNE projections in Figure 5 were based on the CodeBERT features of attack source code in h4rm3l.
    - Clarified qualitative observations on synthesized jailbreak attacks solely expressed in generic primitives.
    - Improved captions for Figures 5. 

- **Discussion and Conclusion:**
    - Included details on synthesizer hparams  $k_{illicit}$, $N_{examples}$ and $N_{proposals}$.
    - Included automated LLM behavior classifier details and adaptability to custom content moderation policies. Addressed **Reviewer CWn6** and **Reviewer fDVc**'s  concerns on how our ASR estimation mechanism can be generalized.
    - Corrected max ASR on GPT-4o to 94\% (vs 90\%)
- **Appendix**
    - *G*: Defense Experiments
        - Corrected table references noted by **Reviewer CWn6**
    - *K*: Generic and Customized Primitives
        - Updated this appendix with usage details of generic decorators, which previously cluttered Section 3.1. This improved presentation also addresses **Reviewer RpnP**'s concerns.
    - *L*: Synthesizer Wallclock Time Performance
        - Added this appendix synthesizer runtime details (timezone redacted to not compromise anonymity).
    - *M*: h4rm3l language specification and implementation details
        - Added this appendix with details of the `PromptDecorator` base class, the h4rm3l program compiler, and the generic primitives. This covers implementation details asked by **Reviewer RpnP**.

We also note the following errata, and pending changes that we will correct in the final manuscript:
- Line 128: "primitives abstractions" corrected to "abstractions" 
- Line 487: "`AffixDecorator` resulted" corrected to "AffixDecorator *could result*"
- Line 217-218: "malicious prompts transformed by a" corrected to "*illicit* prompts transformed by *p*"
- Lines 480-481: "those *of* obfuscation schemes" corrected to "those obfuscation schemes"
- Previously addressed notes from **Reviewer CWn6**'s feedback:
    - In Figure 4, the numbers (e.g. "00536") are synthesized attack identifiers.
    - Acronyms BS, RP, and RT in Tables 3 and 4 mean Baseline, Rephrasing, and Retokenization
    - "proxy names" ( G, B and U?) stand for (Good, Bad and Unclear) Bot behavior.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733295841435

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Expanded Human Evaluation Results

comment:
Based on feedback from **Reviewer CWn6** and **Reviewer fDVc**, we performed an expanded human evaluation on 122 LLM responses uniformly sampled from a dataset of 360 LLM responses (20 responses x 3 classes x 6 LLM).

576 human annotations were performed on those 122 LLM responses. Each model response is annotated by 1 to 7 human annotators following Wei et. al, (2023)'s ternary taxonomy. Their consensus annotation, which is only achieved when there are no ties, is assumed to be the ground truth annotation. This ground truth is used to evaluate both the automated classifier, and the human annotators. The following performance metrics were measured:
- Our zero-shot classifier accuracy is 78\%. Its precision and recall on the BAD BOT class are respectively 72\% and 76\%.
- Humans accuracy is 93\%. Their precision and recall on the BAD BOT class are respectively 91\% and 87\%. 

See details below. We will include these additional results in the final version of our manuscript.

\
\
Classification Performance: Automated Classifier:
==========

```
              precision    recall  f1-score   support

     BAD BOT       0.72      0.76      0.74        38
    GOOD BOT       0.89      0.95      0.92        42
     UNCLEAR       0.70      0.62      0.66        42

    accuracy                           0.78       122
   macro avg       0.77      0.78      0.77       122
weighted avg       0.77      0.78      0.77       122
```

\
\
Confusion Matrix: Automated Classifier:
==========

| Actual↓/Predicted→ | BAD BOT | GOOD BOT | UNCLEAR |
|-------------------|---------|-----------|----------|
| BAD BOT | 29 | 0 | 9 |
| GOOD BOT | 0 | 40 | 2 |
| UNCLEAR | 11 | 5 | 26 |


\
\
Classification Performance: Human Annotators:
==========

```
              precision    recall  f1-score   support

     BAD BOT       0.91      0.87      0.89       164
    GOOD BOT       0.99      0.99      0.99       202
     UNCLEAR       0.89      0.93      0.91       210

    accuracy                           0.93       576
   macro avg       0.93      0.93      0.93       576
weighted avg       0.93      0.93      0.93       576
```

\
\
Confusion Matrix: Human Annotators:
==========

| Actual↓/Predicted→ | BAD BOT | GOOD BOT | UNCLEAR |
|-------------------|---------|-----------|----------|
| BAD BOT | 142 | 1 | 21 |
| GOOD BOT | 0 | 200 | 2 |
| UNCLEAR | 14 | 1 | 195 |
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work proposes a language of primitives, and associated program synthesis approaches for generating new jailbreaks. A variety of results are also provided. All but one reviewer are positive. For the negative reviewer, one of the concerns was the remark "h4rm3l primitives are string-to-string functions written in python that can perform arbitrary computation on strings, and therefore can represent any black-box attack". Strictly speaking, this does not appear to be true, as black-box jailbreaks/red-teaming approaches that allow query access to the LLM are not yet included. The authors should clarify this current limitation in the final version. Weighing the pros and cons, overall this paper merits publication.