Tokenizer-Agnostic Transferable Attacks on Language Models for Enhanced Red Teaming

Authors: Zhakshylyk Nurlanov, Frank R. Schmidt, Florian Bernard
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Adversarial Attacks, Red Teaming, Transferable Attacks, AI Safety, Large Language Models

Abstract:
Large Language Models (LLMs) have become increasingly prevalent, raising concerns about potential vulnerabilities and misuse. Effective red teaming methods are crucial for improving AI safety, yet current approaches often require access to model internals or rely on specific jailbreak techniques. We present TIARA (Tokenizer-Independent Adversarial Red-teaming Approach), a novel method for automated red teaming of LLMs that advances the state-of-the-art in transferable adversarial attacks. Unlike previous token-level methods, TIARA eliminates constraints on gradient access and fixed tokenizer, enabling simultaneous attacks on multiple models with diverse architectures. By leveraging a combination of teacher-forcing and auto-regressive loss functions with a multi-stage candidate selection procedure, it achieves superior performance without relying on gradient information or dedicated attacker models. TIARA attains an 82.9\% attack success rate on GPT-3.5 Turbo and 51.2\% on Gemini Pro, surpassing previous transfer and direct attacks on the HarmBench benchmark. We provide insights into adversarial string length effects and present a qualitative analysis of discovered adversarial techniques. This work contributes to AI safety by offering a robust, versatile tool for identifying potential vulnerabilities in LLMs, facilitating the development of safer AI systems.

Decision: reject
Reviewer scores: 3, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730122110124

thread_type:
official_review

<thread_object>
creator: reviewer_ugpa

<content>
summary:
This paper explores transfer attacks against LLMs. Unlike previous works that utilize gradient to optimize a suitable prompt, this work employs a perturbation-based strategy. The method employs additional models as oracle models and judges the fitness of the current adversarial prompt with the weighted scores of multiple source models. The scoring is based on two loss functions, in which teacher-forcing loss ensures the completion of the target prompt and the auto-regressive loss avoids the occurrence of the deviant tokens.  The attack performance is demonstrated via comprehensive experiments. The attacks against GPT models is impressive.

strengths:
1. **Excellent transfer attack performance.** The reported results are inspiring, especially those against GPT models.
2. **The practicality of the method.** Although an ensemble of multiple models is quite an old talk for universal adversarial attack, it is arguably the most feasible method.

weaknesses:
1. **Inadequate exposure of method details and experiment details.** What about the setting of temperature and top_k for both the auto-regressive loss and evaluation? What about the choice of the subset of candidate tokens that will be sampled mentioned in line 262. After reading the paper, I am still not sure whether the perturbation tokenizer corresponds to the tokenizer of the source model or the one of the target model.
2. **Ad-hoc attack configuration.** The choice of source models is ad-hoc and will impact the attack performance a lot, as mentioned in Section 4.1. Experiments about the lower-bound and upper-bound performance are in demand, corresponding to the weakest source model and the strongest source model.
3. **The analysis of tokenizer behaviors is not in-depth enough.** Although different models use distinct tokenizers, it does not mean they exhibit totally different behaviors. For example, for the same tokenization algorithm, a larger vocabulary of tokenizer A may contain the smaller vocabulary of tokenizer B. Analyzing the overlap between different tokenizer behaviors, especially on the crafted prompts, will help us understand whether the transfer attack leverage semantics or just token combinations to make sense.
4. **Exploration of Attack stability.** As demonstrated in Table 2, the TIARA yields an almost 100\% ASR. As TIARA is in essence an evolutionary method, the stability of the success is amazing. Can the authors give more details about the evolutionary trajectories and analyze the countable failed cases?
5. **Unclear standpoint.** Although the paper emphasizes the tokenizer-agnostic feature of their attack, it is unclear about the difference between tokenizer-agnostic and model-agnostic due to the superficial exploration of the tokenizer behavior. The authors should add a discussion section that clearly defines and distinguishes between tokenizer-agnostic and model-agnostic approaches, and explicitly positions your method within this framework.

I would like to raise my score if the authors can answer my concerns, especially the one about tokenizer behavior (weakness 3).

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1730302765687

thread_type:
official_review

<thread_object>
creator: reviewer_yjp8

<content>
summary:
The paper proposes an algorithm for generating attacks that are highly transferable between open source models and closed source target models. With gradients not being computable on the closed source models (and potentially query numbers being limited), relying on transfer attacks can be a realistic and practical attack method.

strengths:
- The method is empirically effective, simple to implement, and investigates a realistic threat model. Furthermore, with the current perturbation function in the paper we can see that enough random perturbations are sufficient to eventually converge to a jailbreak. I think this is an interesting find that jailbreaks can essentially be brute forced in such a manner on a large scale with sufficient (though not unattainable) compute resources.  

- Using the loss function proposed in the paper combined with an evolutionary search strategy is interesting, and is a different approach compared to popular methods using LLM as a judge to guide evolutionary based strategies (e.g. TAP/PAIR) - potentially this indicates that powerful LLMs as fitness functions in this style of attack may not be needed.

weaknesses:
- It would have been useful too compare with different perturbation functions: the paper shows a random approach, but as highlighted in the work it could be any function. Indeed the random perturbation function is very computationally expensive - 1024 pertubrations per iterations, up with to 1k iterations: in other words, it could be over 1 million separate model calls. Further, additional perturbation functions could highlight if the proposed loss is the most effective under different perturbation strategies.

- The paper flow and balance could be improved: for example, the description of the Multi-Stage Candidate Selection which is a key competent of the algorithm (and is listed as a key contribution of the paper) is in the appendix, and could use a clearer explanation of the steps. 

- It is unclear why Llama3 was used as a transfer model in combination with others, but not used either stand alone, or not evaluated against direct attacks in Table 2.

questions:
- What was the typical number of iterations for an attack success to be found?  Was there a relationship between the longer the optimization and the higher attack transferability?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706036072

thread_type:
official_review

<thread_object>
creator: reviewer_yvoz

<content>
summary:
This paper presents a novel jailbreak method towards LLM based on the attack transferability between the source LLM and the target LLM to achieve tokenzier-agnostic.
Comparing to previous gradient based attack, the attack does not require gradient to optimize jailbreak prompt.
Instead, at each step the attack select best perturbed candicate and use multiple LLM to enhance the attack.
Finally, through experiments, the authors show that the proposed attack outperforms pervious methods and has high attack success rate on commercial LLM like GPT-3.5.

strengths:
+ An effective attack that outperforms previous jailbreak.
+ Evaluation is quite thorough

weaknesses:
- The idea of candicate search has been explored in previous work and the usage of several LLMs to increase the attack transferability is a common technique (which is widely used to increase the adversarial tranferability for image classification models).
Specifically, BEAST attack [1] also investigates how to generate jailbreak prompt without using the gradient. Their beam search-alike method is similar to the TIARA attack here because it essentially selects the potentially best candidates to the target LLM. The differences I see include: TIARA uses multiple source LLM, multiple loss (teacher forcing and autoregressive) and disconnect the tokenizer and source LLM. However, these improvements are not milestone techniques in further improvement jailbreak attack.

- Some design choices, intuition and attack insight, are not clear. For example, how to design the teacher-forcing loss and autoregressive loss? why choose these two loss functions? The use of multiple source LLM can increase the jailbreak attack success rate, but what are the potential reason behind it? There is neither qualitative nor quatitative analysis.

- The tokenizer-agnostic is somehow overclaimed and limited when the target LLM has larger vocabulary. As show in Figure 2, the GPT-4 has lowest ASR comparing to the rest. As the api's tiktoken shows, GPT-4's vocabulary size is much larger than existing opensoured LLMs (which are similar to or better than the GPT-3.5). If future LLM (that should be red-teamed) has larger vocabulary, TIARA may be limited in this case. In other words, my concern is that TIARA may be only effective in red-teaming existing LLMs.

[1] Fast Adversarial Attacks on Language Models In One GPU Minute. ICML 2024.

questions:
1. I wonder the intuition and design insight.
2. How to scale to LLM with tokenizer of much larger vocabulary?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730813033907

thread_type:
official_review

<thread_object>
creator: reviewer_ks9q

<content>
summary:
The paper introduces the Tokenizer-Independent Adversarial Red-teaming Approach (TIARA ), a framework designed to improve AI safety by automating the red teaming of large language models (LLMs). TIARA allows for the generation of transferable adversarial examples without the need for gradient access or fixed tokenizers, facilitating effective attacks across diverse model architectures. Its contributions include a tokenizer-agnostic method for generating adversarial inputs, a gradient-free optimization technique for exploring token-level perturbations, and an automated approach for discovering model vulnerabilities, which aim at identifying potential risks and advancing the development of more secure Chatbot systems.

strengths:
- TIARA presents a tokenizer-independent framework for transferable adversarial attacks, which effectively bypasses constraints associated with fixed tokenization schemes and gradient access, making it adaptable across diverse LLM architectures.
- TIARA employs a multi-stage candidate selection process that iteratively refines adversarial candidates based on both validation and test metrics, maximizing attack success rates across multiple target models.
- TIARA offers a systematic analysis of adversarial strategies, identifying categories such as formatting manipulation and instruction/context manipulation, which advance understanding of adversarial patterns that exploit LLM vulnerabilities.

weaknesses:
- TIARA’s multi-stage candidate selection and extensive perturbation sampling require significant computational resources, making it less feasible for deployment in real-time scenarios.
- While TIARA  is gradient-free, it relies on direct access to model logits for loss computation, which may limit applicability to models where even logit-level access is restricted.
- The paper does not extensively test TIARA against models equipped with advanced, system-level defensive mechanisms, making it unclear how resilient the approach is in more robustly defended environments.
- TIARA lacks an evaluation against input transformation defences, such as synonym replacement or back-translation, which could easily disrupt the token-level perturbations TIARA relies on, potentially reducing its success rate in adversarial bypass attempts.
- TIARA’s reliance on token-level perturbations to generate effective adversarial examples may limit its success on models or tasks where semantic-level manipulations are more impactful. This limitation suggests a need for a more flexible perturbation strategy that balances token- and semantic-level alterations.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733227286801

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal Summary for Reviewers and Area Chairs

comment:
We sincerely thank all reviewers for their detailed evaluations and constructive feedback. Below, we summarize our key contributions, highlight our results, and outline how we addressed reviewer concerns during the rebuttal.

---

### **Key Contributions**

1. **First High-Performance Gradient-Free Attack:**  
   TIARA demonstrates, for the first time, that high attack success rates (ASR) can be achieved through **token-level optimization** without relying on gradient information, hand-crafted prompts, or attacker language models (LLMs). This represents a significant advancement over methods like GCG or TAP, which depend on gradients or external model guidance, and gradient-free baseline BEAST, which achieves insufficient ASR (e.g., 0% on Llama2).  

   TIARA achieves near-perfect ASR in single-model direct attacks on highly secure open-source models:
   - Llama2: **90.2% ASR**  
   - Vicuna: **100% ASR**  
   - Qwen: **100% ASR**  
   - Llama3: **92.7% ASR**

2. **Multi-Model Tokenizer-Agnostic Optimization:**  
   TIARA introduces the first method for **token-level optimization across models with different tokenizers**, enabling effective multi-model attacks. This innovation overcomes significant technical challenges and opens new avenues for studying transferability.

3. **Improved Transferability to Closed-Source Models:**  
   Multi-model attacks consistently improve transferability to closed-source models (Figure 2). For example:
   - Single-model attack on Llama2 achieves **31.7% ASR on Gemini Pro** and **68.3% on GPT-3.5**.  
   - Multi-model attack (Llama2+Vicuna+Qwen) achieves **51.2% ASR on Gemini Pro** and **82.9% on GPT-3.5**.  

4. **Insights into Transferable Attacks:**  
   - Multi-model optimized adversarial strings exhibit **higher token overlap** across tokenizers compared to single-model attacks, reflecting shared vulnerabilities among models.  
   - Multi-model optimized prompts show **better semantic coherence**, indicated by **lower perplexity scores**.  
   - TIARA’s attacks leverage both semantic and token-level strategies—preserving attack intent (e.g., role-playing, contextual manipulation) while benefiting from token-level obfuscations (e.g., formatting and linguistic variations).

5. **Advancing Understanding of Cross-Model Transferability:**  

   We demonstrated that **security by obscurity is insufficient**—a closed-source model alone cannot ensure security. By leveraging transferability, we showed that attacking multiple open-source models significantly enhances attack success rates on closed-source models. Despite these advancements, understanding cross-model transferability of adversarial attacks remains inherently challenging, akin to understanding generalization in machine learning. This difficulty is exacerbated by limited knowledge of target models’ pre-training data and alignment procedures, especially for closed-source systems. Our work provides critical insights and practical methodologies to address this gap, representing a significant step forward in understanding and mitigating this pressing security issue.


---

### **Rebuttal Highlights**
During the rebuttal, we addressed reviewer concerns by providing additional analyses, clarifications, and expanded results, including:  
- **Comparison with BEAST (gradient-free baseline):** TIARA significantly outperformed BEAST, achieving 90.2% vs. 0% ASR on Llama2 and 100% vs. 12.2% on Vicuna.  
- **Added evaluation on Llama3:** We expanded results to include Llama3, achieving 92.7% ASR in direct attacks.  
- **Tokenizer behavior analysis:** We analyzed tokenization behavior on optimized strings and demonstrated higher token overlap and better semantic coherence in multi-model attacks.  
- **Perplexity of transferable strings:** We evaluated the semantic coherence of optimized prompts, showing improved perplexity in multi-model attacks.  
- **Computational cost analysis:** We addressed concerns by reporting the average number of iterations required for the first successful attack across single-model attacks.  
- **Clarification of design choices and terminology:** We elaborated on the intuition behind the algorithm’s design (e.g., loss functions, candidate selection, tokenization choices) and clarified the terms “tokenizer-agnosticity” and “transferability.”

---

### **Conclusion**
TIARA’s contributions—including achieving high ASR without gradients or attacker LLM reliance, tokenizer-agnostic multi-model attacks, and new insights into transferability—constitute a substantial advancement in adversarial attacks on LLMs. During the rebuttal, we addressed all reviewer concerns through rigorous analyses, expanded experiments, and detailed clarifications.

We respectfully request that reviewers reconsider their scores and that these updates be taken into account in the decision-making process.
</content>
</thread_object>

</thread>


Metareview: