A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models (2024)

Abstract: Warning:This paper contains unsafe model responses. Large Language Models (LLMs) have increasingly become central to generating content with potential societal impacts. Notably, these models have demonstrated capabilities for generating content that could be deemed harmful. To mitigate these risks, researchers have adopted safety training techniques to align model outputs with societal values to curb the generation of malicious content. However, the phenomenon of "jailbreaking" — where carefully crafted prompts elicit harmful responses from models — persists as a significant challenge. This research conducts a comprehensive analysis of existing studies on jailbreaking LLMs and their defense techniques. We meticulously investigate nine attack techniques and seven defense techniques applied across three distinct language models: Vicuna, LLama, and GPT3.5 Turbo. We aim to evaluate the effectiveness of these attack and defense techniques. Our findings reveal that existing white-box attacks underperform compared to universal techniques and that including special tokens in the input significantly affects the likelihood of successful attacks. This research highlights the need to concentrate on the security facets of LLMs. Additionally, we contribute to the field by releasing our datasets and testing framework, aiming to foster further research into LLM security. We believe these contributions will facilitate the exploration of security measures within this domain.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 7432–7449 August 11-16, 2024 ©2024 Association for Computational Linguistics
Large Language Models (LLMs) have increasingly become central to generating content with potential societal impacts. Notably, these models have demonstrated capabilities for generating content that could be deemed harmful. To mitigate these risks, researchers have adopted safety training techniques to align model outputs with societal values to curb the generation of malicious content. However, the phenomenon of "jailbreaking" — where carefully crafted prompts elicit harmful responses from models — persists as a significant challenge. This research conducts a comprehensive analysis of existing studies on jailbreaking LLMs and their defense techniques. We meticulously investigate nine attack techniques and seven defense techniques applied across three distinct language models: Vicuna, LLama, and GPT3.5 Turbo. We aim to evaluate the effectiveness of these attack and defense techniques. Our findings reveal that existing white-box attacks underperform compared to universal techniques and that including special tokens in the input significantly affects the likelihood of successful attacks. This research highlights the need to concentrate on the security facets of LLMs. Additionally, we contribute to the field by releasing our datasets and testing framework, aiming to foster further research into LLM security. We believe these contributions will facilitate the exploration of security measures within this domain.

heading: 1 Introduction
text: Large Language Models (LLMs), such as GPT (OpenAI, 2023b) and LLaMa (Hugging Face, 2023a), play a pivotal role across a spectrum of applications, from text summarization (Tian et al.,
Corresponding Authors.
2024) to code generation (Ni et al., 2023). The popularity of LLMs in everyday scenarios underscores their significance. However, this ubiquity also raises security concerns associated with LLMs (Ouyang et al., 2022).
Several types of vulnerabilities have been identified in LLMs (OWASP, 2023). Among these, the jailbreak attack stands out as a prevalent vulnerability, where specially designed prompts are used to bypass the safety measures of LLMs, facilitating the production of harmful content. There has been notable research aimed at addressing jailbreak attacks. For example, Liu et al. (Liu et al., 2023b) investigate various mechanisms for jailbreak prompting and assess their effectiveness. Zou et al. (Zou et al., 2023) apply a white-box approach combined with adversarial attacks to create jailbreak prompts. Additionally, Deng et al. (Deng et al., 2023a) explore using LLMs to generate jailbreak prompts in a black-box setting. To defend against jailbreak attacks, Robey et al. (Robey et al., 2023) proposed a method that involves randomly omitting a certain number of tokens from the input to detect malicious attempts. Meanwhile, Pisano et al. (Pisano et al., 2023) introduced an approach that employs an auxiliary model to assist the primary model in identifying hazardous information.
Despite the various jailbreak attack and defense methodologies, to the best of our knowledge, there remains a significant gap in the literature regarding comprehensive evaluations of how well the attack methodologies can perform against defended LLMs and how well defense mechanisms against jailbreak attacks. While Mazeika et al. (2024) and Zhou et al. (2024) explore various attack techniques, they did not evaluate those on defense techniques, and vice versa.
To address this research gap, we undertake a comprehensive empirical study on jailbreak attack and defense techniques for LLMs. Our study is designed to answer two critical research ques-
7432
tions. First, we investigate the effectiveness of various jailbreak attack approaches on different unprotected LLMs, encapsulated in the question (RQ1: Effectiveness of Jailbreak Attacks). Second, we evaluate the effectiveness of defense strategies against these attacks on varied LLMs, posed as (RQ2: Effectiveness of Jailbreak Defenses).
During the Baseline Selection phase, we chose nine attack methods and seven defense mechanisms, drawing on four seminal works, including notable libraries (Automorphic, 2023; ProtectAI, 2023), and the OpenAI Moderation API (OpenAI, 2023), prioritizing prevalent and accessible methods with open-source code.
In the Benchmark Construction phase, our benchmark, initially based on (Liu et al., 2023b), was expanded through additional research (Zou et al., 2023) and a GPT model in "Do Anything Now" mode, resulting in 60 categorized malicious queries following OpenAI’s guidelines.
For Result Labeling, the RoBERTa model was fine-tuned for classifying malicious responses, achieving 92% accuracy, outperforming GPT-4’s 87.4%. Manual validation ensured the reliability of our classification.
In the Evaluation Phase, we employed metrics for assessing attack efficiency and effectiveness, alongside defense robustness against malicious and benign inputs, establishing a comprehensive framework for evaluating LLM security.
Our analysis reveals several notable insights. Specifically, among the various jailbreak attack techniques, template-based methods demonstrate superior effectiveness. In contrast, gradient-based generative approaches, especially in ’white-box’ scenarios, generally fall short of the performance achieved by universal generative methods. Additionally, our findings highlight the significant impact of special tokens on the success probability of attacks. As for defense techniques, we identify the Bergeron method as the most effective defense strategy to date, while all other defense techniques in our study perform badly as they either cannot stop jailbreak attacks at all or are too strict such that benign prompts are also prohibited. Our results underscore a great need for the development of more robust defense mechanisms.
In summary, our work presents several contributions to the field:
• Comprehensive Study. This study represents, to the best of our knowledge, the first systematic evaluation of the effectiveness of jailbreak attacks versus defenses on various open/closedsource LLMs.
• Key Findings. We uncover previously unknown insights that hold significant potential for enhancing both attack and defense strategies in the future.
• Open-source Artifacts. We develop and publicly release the first benchmark that includes a comprehensive collection of both attack and defense techniques, thereby facilitating further research in this area.
The raw data, the benchmark platform, and additional details are available on a companion website of this paper: https://sites.google.com/view/ llmcomprehensive/home.

heading: 2 Background and Related Work
text: This study underscores the effectiveness of specific attack methodologies against various defense strategies and vice versa, filling a gap not addressed in contemporary literature (Mazeika et al., 2024; Zhou et al., 2024). These works primarily focus on evaluating various attack techniques against unprotected models, with the exception of initial safety training. Our research conducts the first comprehensive survey that evaluates the reciprocal impacts of both attack and defense techniques.

heading: 2.1 LLM Jailbreak
text: Jailbreak attacks on LLMs involve crafting prompts that exploit the models to generate malicious content. Despite the potential for harm, such as generating instructions for fabricating explosives, LLMs typically refrain from producing such responses due to the incorporation of safeguards during their training. These measures include Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022), Robustness via Additional Fine-Tuning (RAFT) (Dong et al., 2023), and Preference Optimized Ranking (PRO) (Song et al., 2023), which ensure the model’s adherence to ethical guidelines.
The precise mechanisms behind the jailbreak phenomena remain under debate. Wei et al. (Wei et al., 2023) postulate that jailbreaks may occur in scenarios where safety training is insufficiently comprehensive, allowing for the generation of content in unmonitored areas, or when the model encounters dilemmas between providing useful responses and maintaining safety protocols. Complementing this, Subhash et al. (Subhash et al., 2023) explored the role of the model’s hidden states in gradient-based attacks, identifying that a specific suffix, when appended to the original prompt, serves as an embedding vector guiding the model toward generating inappropriate content. This finding aligns with the hypothesis that jailbreaks can manifest in regions not fully covered by safety training, enabling the production of objectionable content.
“Benign content” is defined as responses considered morally or ethically inappropriate, with OpenAI compiling an extensive list of such categories. Liu et al. (Liu et al., 2023b) further elaborate on this classification, providing a framework for categorizing these responses. The assessment presented herein conforms to this established categorization, ensuring a structured approach to understanding and mitigating jailbreak risks in LLMs.
In the subsequent subsection, we present a categorization of current attack and defense techniques. Additionally, we analyze the pros and cons of each category in various dimensions. Details can be found in Appendices A.1 and A.2. This analysis facilitates a comprehensive understanding and substantiates our categorization approach.

heading: 2.2 Jailbreak Attack Techniques
text: To provide a structured overview of the strategies utilized to compromise LLMs, we categorize current attack techniques into three categories, reflecting their fundamental traits. The first category, Generative Techniques, includes attacks that are dynamically produced, eschewing predetermined plans. The second category, Template Techniques, comprises attacks conducted via pre-defined templates or modifications in the generation settings. The last category, Training Gaps Techniques, focuses on exploiting weaknesses due to insufficient safeguards in safe training practices, such as RLHF (Ouyang et al., 2022). The techniques employed in our study are elaborated in Table 1, highlighting the methods chosen for evaluation within our framework.

heading: 2.3 Jailbreak Defense Techniques
text: We further conduct a thorough examination of the existing defense mechanisms, classifying them into three categories based on their operational principles: Self-Processing Defenses, which rely exclusively on the LLM’s own capabilities; Additional Helper Defenses, which require the support of additional algorithms or auxiliary LLMs for verification purposes; and Input Permutation Defenses, which manipulate the input prompt and verify with the target LLMs multiple times to detect and counteract malicious requests aimed at exploiting gradient-based vulnerabilities. An overview of these defense mechanisms is presented in Table 2.

heading: 3 Study Design
text: Our study aims to address two core research questions: RQ1 (Effectiveness of Jailbreak Attacks): How effective are jailbreak attack techniques across various LLMs? RQ2 (Effectiveness of Jailbreak Defenses): How effective are jailbreak defense techniques against various attack techniques when protecting different LLMs?

heading: 3.1 Baseline Selection
text: Our methodology selection criteria were predicated on the method’s popularity and accessibility to source code. For RQ1, our analysis covers nine attack techniques, divided into five generative (AutoDAN (Liu et al., 2023a), PAIR (Chao et al., 2023), TAP (Mehrotra et al., 2023), GPTFuzz (Yu et al.,
2023), GCG (Optimize per prompt on a single model) (Zou et al., 2023)) and four template-based approaches (Jailbroken (Wei et al., 2023), 78 Templates from existing study (Liu et al., 2023b), Deep Inception (Li et al., 2023a), Parameters (Huang et al., 2024)). To elucidate the characteristics of the prompts used in attack techniques, we present an illustrative example in Figure 7.
For RQ2, we examine four defense techniques: Bergeron (Pisano et al., 2023) and Baseline (Jain et al., 2023) for additional helper methods; RALLM (Cao et al., 2023) and SmoothLLM (Robey et al., 2023) for input permutation techniques; Notable open-source projects, Aegis (Automorphic, 2023) and LLMguard (ProtectAI, 2023), alongside the OpenAI Moderation API (OpenAI, 2023), are also evaluated for their defense efficacy. Limitations such as Rain’s (Ouyang et al., 2022) extensively prolonged time-consuming processing and Certifying-llm’s (Kumar et al., 2023) scalability issues are considered to be excluded from our selection.

heading: 3.2 LLMs under Test
text: In our research, we focus on evaluating three distinguished models: Llama-2-7b (Hugging Face, 2023a), Vicuna-v1.5-7b (Hugging Face, 2023b), and GPT-3.5-Turbo-1106 (OpenAI, 2023b). These models were chosen due to their prevalent use in security-related research, encompassing both attack simulations and the development of defensive strategies. The decision to omit GPT-4 from our evaluation stems from its significant operational requirements. Preliminary evaluations of GPT-3.5Turbo revealed an exceptionally high query count, totaling 79,314. When taking into account the economic ramifications associated with the token pricing of GPT-4, which is established at $0.01 per 1,000 tokens (OpenAI, 2023a), this financial consideration renders the incorporation of GPT-4 into a comparative study economically challenging.

heading: 3.3 Experimental Configuration
text: Our experimental framework utilized two NVIDIA RTX 6000 Ada GPUs, each outfitted with 48 GB of RAM. We aligned our testing parameters with those identified as optimal in the relevant litera-
ture, defaulting to the original repositories’ settings in the absence of specific recommendations. To address RQ1 and ensure consistency across different attack methodologies, each query was executed 5 times to minimize variability. For the evaluation involving generative models, we capped the process at a maximum of 75 iterations for each query, defining an iteration as a single algorithmic step. However, our empirical study of GCG with 18 questions that were randomly and uniformly sampled from six categories suggests that GCG only on Llama requires a higher number of iterations to jailbreak most queries; otherwise, failure. In order to not be biased to GCG, we use the default 500 iterations on the Llama model only. We provide a further discussion in Section 6.1

heading: 3.4 Benchmark Construction
text: We leveraged the benchmark framework proposed by Liu et al. (Liu et al., 2023b). This benchmark is distinguished by its rigorous focus on policy compliance to OpenAI categories(OpenAI, 2023) within the context of malicious content detection. In an effort to enhance the robustness of our evaluation, we expanded the original dataset to include 60 malicious queries, effectively doubling its size. This augmentation was achieved through meticulous manual curation and integrating selected examples from AdvBench (Zou et al., 2023). Our approach to dataset expansion adhered strictly to the categorization and selection criteria established in previous studies, ensuring both the consistency and the relevance of the enhanced dataset for comprehensive evaluation.

heading: 3.5 Result Labeling
text: In our study, we employed both automated and manual labeling strategies to categorize the responses gathered from our evaluation process, details can be found in Appendix A.3.

heading: 3.6 Evaluation Metric
text: For RQ1, we use two metrics. This dual metric approach ensures a comprehensive evaluation of both the attack’s impact and its operational feasibility. First, Attack Success Rate (ASR): defined as the ratio of successfully compromised questions c to the total number of questions n, ASR measures the effectiveness of an attack.
ASR = c
n . (1)
Second, Efficiency: this metric quantifies the effectiveness of attack queries, defined as the ratio of the number of individual queries q that successfully compromise the model to the total number of query attempts o. Each query represents a minimal experimental unit or a single prompt.
Efficiency = q
o . (2)
For RQ2, we introduce three metrics that ensure a balanced assessment of system robustness and output integrity. The first, Defense Passing Rate (DPR), calculates the ratio of prompts f that incorrectly bypass the defense mechanism—being erroneously classified as harmless—to the total number of malicious inputs m.
DPR = f
m . (3)
The second metric, Benign Success Rate (BSR), assesses the proportion of non-malicious inputs s that successfully navigate through the defense filter relative to the total number of inputs t.
BSR = s
t . (4)
Lastly, the Generated Response Quality (GRQ) evaluates the quality of responses generated by defense mechanisms compared to a standard reference. To assess the responses to benign queries, we employ the Alpaca Eval framework (Li et al., 2023b), leveraging its methodology for automatically evaluating response quality. Evaluating GRQ is crucial for methodologies that produce new responses (Cao et al., 2023; Robey et al., 2023; Pisano et al., 2023).

heading: 4 RQ1: Effectiveness of Jailbreak Attack
text: The effectiveness of attack strategies on the selected LLMs under test is systematically presented in Tables 6, 7, and 8. To offer a clearer comparative analysis of model performance, we consolidated these metrics into a scatter plot depicted in Figure 2. In this visualization, models demonstrating optimal performance are positioned nearer to the scatter plot’s upper right quadrant, signifying superior ASR and Efficiency.
Evaluation results reveal that using 78 templates, Jailbroken, and GPTFuzz strategies yield superior results in circumventing the security measures of GPT-3.5-turbo and Vicuna. Conversely, for LLaMA, strategies such as Jailbroken, Parameter,
and 78 templates demonstrated the highest effectiveness. This prevalence of template-based approaches highlights their efficiency, primarily due to the intricate design of their prompts. The most successful five templates from these strategies are listed in Table 16.
In the realm of generative strategies, GPTFuzz, Pair, and Tap emerged as the top performers. Moreover, it was noted that LLaMA presents a noteworthy challenge for jailbreaking compared to Vicuna. We will discuss this in Section 6.1. Additionally, our study into the categories of questions that were successfully jailbroken indicates that queries related to unlawful practice, harmful content, and illegal activities are the most challenging to address across all tested models. Details can be found in Tables 9, 11, and 10.

heading: 5 RQ2: Effectiveness of Jailbreak defense
text: Our study meticulously evaluates defense mechanisms against malicious queries as well as the handling of benign questions. The outcomes of this evaluation are systematically tabulated in Tables 12, 13, and 14. These results are further visualized in Figure 3, where the optimal defense strategies are identified by their proximity to the upper left corner of the plot, signifying lower DPR and higher BSR. Our findings reveal that, apart from the Bergeron method, the efficacy of the current defense strategies remains largely inadequate. Additionally, our comparative analysis of the quality of benign responses generated through three innovative methodologies disclosed minor variance among them, as elaborated in Table 15.

heading: 6 Discussion

heading: 6.1 Comparative Performance of White-Box and Black-Box Attacks
text: Our investigation reveals that white-box attacks are less effective than black-box jailbreak strategies. Specifically, methods like AutoDan and GCG, which rely on insights into the model’s internal mechanisms, such as loss metrics, underperform when compared to universal, template-based attack methods that do not necessitate access to a model’s internals and are pre-designed. Moreover, the LLaMa model presents more significant challenges to jailbreaking efforts, particularly under white-box attack strategies, than Vicuna. This observation is intriguing, especially considering that Vicuna is an evolution of LLaMa, having been refined through additional fine-tuning processes (LMSYS, 2023). The pronounced resilience of LLaMa against attacks highlights the critical role of comprehensive safety training during its development phase, suggesting that such training is a crucial element in bolstering the defenses of open-source LLMs.
To further understand the influence of loss metrics on a model’s vulnerability to jailbreaking, we conducted a targeted experiment. A question was randomly selected from our dataset, and the experiment’s findings are visually represented in Figure 4. The experiment showed that Vicuna began the process with a higher initial loss but saw a significant reduction in loss, stabilizing after 12 steps and five successful jailbreak attempts. However, it maintained a higher final loss compared to LLaMa. In
contrast, LLaMa started with a lower initial loss and demonstrated a slower reduction in loss over time, ultimately failing to jailbreak the question within 75 iteration steps despite exhibiting a significantly lower final loss than Vicuna. These outcomes suggest that LLaMa’s foundational safety training plays a pivotal role in its enhanced defense against jailbreak attempts. It implies that integrating advanced safety training protocols into developing open-source models could markedly reduce the efficacy of white-box attacks, thereby enhancing their security posture.

heading: 6.2 Impact of Special Tokens on Jailbreak Attack Performance
text: Our research has uncovered that using special tokens significantly influences the success rates of jailbreak attack techniques. Specifically, the deployment of 78 templates on GPT-3.5-Turbo and Vicuña models has spotlighted the substantial effect of the special token ‘
’ on compromising the LLaMa model. Through methodical experimentation with these templates, as systematically documented in Table 3, we sought to understand the differential impact of various configurations on attack effectiveness.
The analysis focused on four distinct settings, leading to the identification of five templates that demonstrated the most significant disparities in performance, detailed in Table 4. Notably, we discovered that text continuation templates were rendered ineffective by the inclusion of ‘
’, rationalized by the implication that the absence of this token signals an intent for incomplete input, thereby encouraging the model to supplement the provided prompt before generating a response. This pattern was consistent across multiple templates, showing
that the model often elaborates on the user’s initial prompt, though not invariably resulting in a successful attack.
Particularly interesting was the observation that omitting ‘
’ from two specific templates reduced the probability of a successful jailbreak. These templates predominantly involved scenarios of character role-play, as characterized in previous studies (Liu et al., 2023b). The underlying reasons for this discrepancy are not immediately apparent, suggesting a potential misalignment possibly due to gaps in the model’s safety training.
Given these preliminary findings, there is a compelling need for more comprehensive investigations into the influence of special tokens on the behavior of LLMs in the context of security vulnerabilities. Future research should extend beyond ‘
’ to explore the effects of other special tokens, such as ‘«sys»’ and ‘
’ on LLaMa and additional models. Such studies promise to deepen our understanding of model responses to crafted inputs and inform the development of more robust defense mechanisms against jailbreak attacks.

heading: 6.3 Enhancing Defense Mechanisms Against Diverse Malicious Queries
text: Our research underscores the imperative need for robust defense mechanisms to be rigorously evaluated against a broad spectrum of malicious queries. The development of a uniform and standardized evaluation methodology emerges as a primary challenge within the realm of defense strategies. Present techniques, as outlined by Kumar et al. (2023), Robey et al. (2023), and Cao et al. (2023), rely on predefined sets that frequently misclassify benign responses, such as "I’m sorry, I do not understand," as malicious. This issue of misclassification becomes even more pronounced when incorporating models like GPT-4 with these sets, as GPT-4’s limitations in generating contextually accurate responses can lead to benign inputs being wrongly identified as malicious, as illustrated in Figure 8.
The lack of a uniform evaluation methodology is also reflected by the challenge, Detection Constraints. Although (OpenAI, 2023) categorizes several forms of toxic content, the diversity in expressions and responses can overwhelm sophisticated classifiers, as evidenced by the performance of the ModerationAPI. Moreover, the efficacy of detection models is inherently limited by their model characteristics, such as the number of parameters,
model structure and training dataset, etc.
Moreover, our analysis reveals that another challenge, Cost Concerns. While the Bergeron method shows promise, its effectiveness is limited by its dependence on LLMs and the use of predefined prompts for processing and verification, which not only undermines its reliability but also entails substantial operational costs.
Another concern is Latency Issues. In our experiments, we observed that references RALLM (Cao et al., 2023) and Smooth-LLM (Robey et al., 2023) exhibit significantly longer processing times for queries and struggle with scalability issues, particularly when dealing with complex natural language inputs. This issue is attributable to their design approach, which involves repeatedly checking a single prompt variation, consequently prolonging the defense time. A similar issue was also noted in our preliminary experiments with RAIN (Li et al., 2023c).
Given these observations, there is a critical and pressing need for further research into more advanced evaluation frameworks and formulating more effective defense strategies. Such efforts should aim to circumvent the current challenges by ensuring reliable differentiation between malicious and benign inputs across varying contexts and increasing the scalability of defense mechanisms to accommodate the complexities inherent in natural language processing.

heading: 7 Conclusions
text: In this work, we present the first comprehensive assessment of existing attack and defense strategies in the context of LLM security. Additionally, we contribute to the field by releasing the first framework specifically designed for assessing the robustness of LLMs against various threats. We selected nine attacks and seven defensive mechanisms from existing literature and software libraries for our analysis. Our experimentation, conducted on three distinct models, reveals that Template methods are notably effective, with 78 templates technique identified as the most powerful one. Regarding Generative methods, GPTFuzz emerged as the most effective given the experiment budget. Our investigation into question categorization demonstrated that all three models exhibit enhanced resilience against queries related to unlawful practice, harmful content, and illegal activities. However, our analysis of current defensive measures indicates a general ineffectiveness, with Bergeron showing comparatively better performance. We highlight the necessity of establishing a uniform baseline for jailbreak detection, as existing defenses employ varied methodologies, and the need to develop better defense techniques. Additionally, our study observed the impact of using the ’
’ marker in the Llama model. Looking forward, we aim to continuously incorporate evolving attacks and defenses into our framework, thereby providing a dynamic overview of the field’s progression.

heading: 8 Limitations
text: To address the constraints posed by limited resources, our evaluation does not extend to larger models, such as those with 13 billion and 33 billion parameters, nor does it cover powerful models like GPT-4 and other commercial models, including Gemini (Gemini) and Palm2 (AI). Regarding autoDan, it is noteworthy that significant updates were identified in its repository as of February 2024. Given that our evaluation was completed prior to these updates, the outcomes may be impacted. Nonetheless, we intend to align our repository with these recent modifications soon.

heading: 9 Ethical Considerations and Disclaimer
text: In conducting this study, our research team has committed to the highest standards of ethical conduct by exclusively utilizing resources that are publicly accessible. We have undertaken this research with a conscientious commitment to ethical principles, ensuring that all of our activities are aligned with the established norms and guidelines of responsible scientific inquiry.
Aware of the fine line between knowledge advancement and safety assurance, we introduced measures like limiting the length of potentially malicious responses in our dataset. This method aims to support evaluation and learning without revealing practical information prone to misuse. We emphasize our dedication to ethical practices by actively reducing the risk of spreading harmful content.
In the spirit of transparency and accountability, we have taken proactive steps to ensure that all of our findings are managed with the utmost responsibility. This includes the systematic reporting of our results to the developers and providers of the models we have analyzed. Our aim is to contribute constructively to the ongoing dialogue regarding the security of LLMs and to aid in the identification and mitigation of potential vulnerabilities.

heading: A APPENDIX
text: A.1 Analysis of Categorization of Attack Techniques
Our empirical analysis and experimental results identified five metrics for assessing the advantages and disadvantages of various attack techniques, as shown in Figure 5.
The criterion of Complexity measures the intrinsic algorithmic challenge posed by each method. Notably, the Generative approach is identified as the most complex, attributed to its sophisticated algorithmic underpinnings. This is followed by the Training Gaps method, which demands substantial insight into the model’s operation for effective application.
The dimension of Specificity evaluates whether an attack is tailor-made for a particular model.
Given that Training Gaps are dependent upon the unique safety training protocols of each model, they inherently exhibit the highest specificity. Subsequently, the Template-Based method, often crafted for specific model types (e.g., the GPT series), ranks next in specificity.
In terms of Ease of Use, the Template-Based approach emerges as the most user-friendly, attributed to its pre-designed nature, thereby facilitating immediate application. The Training Gaps method follows, offering relatively straightforward deployment when contrasted with the more complex Generative approach.
Regarding Ease of Fix, Template-Based attacks, due to their predefined structure, allow for direct incorporation into safety training protocols, simplifying mitigation efforts. Similarly, addressing vulnerabilities exposed by Training Gaps is comparatively easier.
Lastly, the criterion of Running Cost reveals that Generative techniques, due to their intensive iteration and deployment requirements, incur the highest expenses. The Template-Based method, necessitating the processing of extensive prompts, ranks second, surpassing Training Gaps in terms of token processing demands.
A.2 Analysis of Categorization of Defense Techniques
Similarly, we designed four key metrics for evaluating the pros and cons of different defense techniques.
Autonomy assesses the degree to which a model depends on external resources for detection. SelfProcessing exhibits the highest autonomy, followed by Input Permutation—this method necessitates further processing of the input and multiple generations with the LLM. Conversely, the Additional Helper method employs an external algorithm or model for decision-making.
Running Cost evaluates the operational expenses; Input Permutation is notably resourceintensive due to significant input modifications and subsequent model verifications, making it more expensive than Additional Helper, followed by SelfProcessing. However, Rain (Li et al., 2023c) of Self-Processing category presents an outlier, incurring prolonged processing times due to its autoregressive input examination.
Adaptability assesses the capability of the approach to adjust to emerging attack vectors. In light of the continuous progress and enhancement of detection algorithms in LLMs, the Additional Helper strategy, typically involving the integration of a further sophisticated model or algorithm, gains from persistent updates. Likewise, the Self-Processing approach benefits from this adaptability.
Lastly, Comprehensiveness measures a defense mechanism’s capability to generalize across attack types. Employing Additional Helper specializing in identifying malicious inputs ranks highest in effectiveness, followed by Self-Processing, which is inherently limited by the model’s capabilities, and input permutation, which usually disrupts the attacks utilizing the embedded gradient information such as GCG (Zou et al., 2023).
A.3 Customized Evaluator With a dataset encompassing 199,427 responses, the task of identifying malicious content posed significant challenges due to the impracticality of manual review and the absence of standardized evaluation methods for malicious responses—a gap in current research.
Existing strategies for addressing this issue vary. Zou et al. (2023) utilized a set of common refusal patterns, such as "I am sorry" and "I cannot", to automate the identification of non-compliant responses. Yu et al. (2023) and Huang et al. (2024) focused on enhancing machine learning models, specifically RoBERTa and BERT-BASE-CASED models, respectively. Additionally, approaches leveraging GPT-4 for attack analysis were explored by Chao et al. (2023) and Mehrotra et al. (2023).
To ensure a rigorous analysis of our dataset, we selected a random sample of 1,068 responses, aiming for a 95% confidence level with a 3% margin of error. This sample underwent manual annotation to assess each response’s relevance to the inquiry and its alignment with providing a malicious answer. An example of this annotation process is documented in Figure 9, and a comparative analysis of the effectiveness of different models is provided in Table 5.
Reflecting on the reported efficacy of the finetuned RoBERTa model by Yu et al. (2023), we chose to refine this model further utilizing our manually annotated dataset, accessible on HuggingFace (fine tuned, 2024). The fine-tuning protocol involved a batch size of 5, three training epochs, a learning rate of 2× 10−5, application of the Adam optimizer, and a linear rate decay complemented by a warm-up phase covering 10% of the training duration. Post-labeling, an additional round of random sampling was conducted for manual verification to ascertain the accuracy and reliability of our findings.
Table 15: Evaluation of benign response quality for Alpaca using Smooth, Rallm, and Bergeron, where higher levels signify improved quality.
Model Smooth GRQ Rallm GRQ Bergeron GRQ GPT-3.5-Turbo 9.13 7.89 8.01 Vicuna 2.24 0.75 4.78 Llama-2 6.21 5.47 5.65
Figure 8: An example of false positive classification of safe response to unsafe of GPT-4

heading: Future Work
text: Future research should extend beyond ‘
’ to explore the effects of other special tokens, such as ‘«sys»’ and ‘
’ on LLaMa and additional models. Such studies promise to deepen our understanding of model responses to crafted inputs and inform the development of more robust defense mechanisms against jailbreak attacks