Venue: NAACL 2025

# SeqAR: Jailbreak LLMs with Sequential Auto-Generated Characters

 Yan Yang\({}^{1*}\), Zeguan Xiao\({}^{1*}\), Xin Lu\({}^{3*}\), Hongru Wang\({}^{4}\), Xuetao Wei\({}^{2}\)

Hailiang Huang\({}^{1\dagger}\), Guanhua Chen\({}^{2\dagger}\), Yun Chen\({}^{1,5\dagger}\)

\({}^{1}\)Shanghai University of Finance and Economics

\({}^{2}\)Southern University of Science and Technology

\({}^{3}\)Ant Group, \({}^{4}\)The Chinese University of Hong Kong

\({}^{5}\)MoE Key Laboratory of Interdisciplinary Research of Computation and Economics,

Shanghai University of Finance and Economics

Equal Contribution.Corresponding Authors.

###### Abstract

The widespread applications of large language models (LLMs) have brought about concerns regarding their potential misuse. Although aligned with human preference data before release, LLMs remain vulnerable to various malicious attacks. In this paper, we adopt a redteaming strategy to enhance LLM safety and introduce SeqAR, a simple yet effective framework to design jailkreak prompts automatically. The SeqAR framework generates and optimizes multiple jailbreak characters and then applies sequential jailkreak characters in a single query to bypass the guardrails of the target LLM. Different from previous work which relies on proprietary LLMs or seed jailkreak templates crafted by human expertise, SeqAR can generate and optimize the jailkreak prompt in a cold-start scenario using open-sourced LLMs without any seed jailkreak templates. Experimental results show that SeqAR achieves attack success rates of 88% and 60% in bypassing the safety alignment of GPT-3.5-110G and GPT-4, respectively. Furthermore, we extensively evaluate the transferability of the generated templates across different LLMs and heldout malicious requests, while also exploring defense strategies against the jailkreak attack designed by SeqAR. Warning: This paper contains unfiltered content generated by LLMs that may be offensive to readers.

## 1 Introduction

Large language models (LLMs), such as ChatGPT [7] and LLaMA [9], have achieved impressive performance in different natural language processing tasks in recent days. LLMs advance neural-based applications in real-world applications with their superior text understanding, generation, and reasoning abilities. After finetuning with instruction data, the LLMs enhance their instruction-following ability to respond to diverse input commands [14]. However, the remarkable instruction-following ability brings about significant risks of potential LLM misuse [14]. For example, when prompted with malicious instructions, the LLM may directly give some suggestions about how to spread a rumor. Therefore, it is imperative to align deployed LLMs with human values in real-world applications. The LLMs are aligned with human preference data using methods like Reinforcement Learning via Human Feedback [14], RLHF) and Direct Preference Optimization [15, DPO]. The aligned LLMs are expected to generate helpful, harmless, and honest responses.

However, aligned proprietary LLMs are still attacked to generate harmful content in some scenarios [13, 12]. They remain vulnerable to adversarial inputs, particularly in the context of jailkreak attacks [15]. Jailbreak attempts to intentionally craft user instructions that trigger LLMs to generate harmful, offensive, or undesirable content in accordance with the malevolent user's intention [12, 14]. Red-teaming is a strategic approach to enhance LLM safety by actively investigating and revealing hidden scenarios

Figure 1: A simplified example of jailkreaking LLMs with SeqAR.



in which LLMs may exhibit failures (Perez et al., 2022; Ganguli et al., 2022). Previous researches enhance LLM safety with a red-teaming strategy by designing jailbreak prompts either by hand or automatically (Zou et al., 2023; Zhu et al., 2024; Chao et al., 2023; Li et al., 2024). However, the hand-crafted methods are hardly scalable to different LLMs. The optimization-based methods suffer from unsatisfied attacking performance, especially for the iteratively updated proprietary LLMs.

In this paper, we propose SeqAR, a framework to jailbreak LLMs with **SEQ**uential **A**uto-generated cha**R**acters. It is a simple and effective framework that can design and optimize the jailbreak templates automatically. Inspired by previous attempts that exploit role-playing to bypass the safety guardrails of LLMs (DAN, 2023; Li et al., 2024) and research on the distraction phenomenon in LLMs (Shi et al., 2023; Xiao et al., 2024), we apply **sequential** jailbreak characters within a single user query and ask the target LLMs to act as these characters sequentially to provide step-by-step instructions (see the example in Figure 2). LLMs are notably fragile and easily distracted by our complex role-play instructions Hu et al. (2024); Wang et al. (2024); Chen et al. (2024), thereby enhancing the attack performance of each individual jailbreak character. Furthermore, since different characters have different attack scopes, their combination can yield a more comprehensive and effective attack capability. These sequential characters are automatically optimized starting from a very simple character with the help of the attacker LLM. During optimization, a sequence of characters is determined greedily to maximize the best attack performance of character combinations at each step.

We compare SeqAR with baselines on the AdvBench custom dataset. With LLaMA-2-7B-chat as an attacker LLM, SeqAR successfully jailbreaks LLaMA-2-7B-chat (Touvron et al., 2023), LLaMA-3-8B-Instruct (Dubey et al., 2024) , GPT-3.5-0613 (OpenAI, 2022), GPT-4o (OpenAI, 2024), GPT-4 (OpenAI, 2023) and Gemini-1.5-Pro (Reid et al., 2024) with an ASR score of 92%, 40%, 86%, 50%, 60% and 78%, respectively, which significantly surpasses the baseline of PAIR (Chao et al., 2023), GPTFuzzer (Yu et al., 2023) and PAP (Zeng et al., 2024). The attack performance can be further improved to 84% ASR on GPT-4 when combined with long-tail encoding method. We conduct extensive ablation studies and analyses to demonstrate the necessity of sequential character-playing, the cross-request and cross-target model transfer capacity of SeqAR, and the effectiveness of various defense methods against SeqAR.1

Footnote 1: Our code is publicly available at [https://github.com/sufenlp/SeqAR](https://github.com/sufenlp/SeqAR).

## 2 Methods

SeqAR is a black-box jailbreak attack framework (see Figure 2) for automated red-teaming of LLMs even where only APIs are accessible. Sequential jailbreak characters (Section 2.1) are automatically generated and optimized (Section 2.2) under the judgement model (Section 2.3) to work together for jailbreaking the target LLM.

### Jailbreak via Sequential Characters

There are many attempts to exploit role-playing to bypass the intrinsic safety guardrails of LLMs.

Figure 2: The overview of SeqAR framework. (Left): The jailbreak attack template of SeqAR. The character descriptions are generated and optimized in a greedy manner. During character optimization, the r\({}^{\text{th}}\) character is generated and evaluated with the previous r-1 characters fixed. The model response starts with these sentences . (Right): The automated generation and optimization process of the r\({}^{\text{th}}\) jailbreak character. The meta prompt is used to generate character descriptions with an attacker LLM. More details are in Section 2.2.

 

These attempts encompass both manually designed characters (e.g., DAN [4]), AIM2) and automatically generated ones [10]. The success of these role-playing jailbreak attempts can be attributed to two primary factors. First, there exists a conflict between LLMs' instruction-following training and their safety training, i.e., "competing objectives" [35]. LLMs tend to follow it when instructed to act as a specific character since it is harmless-looking. Second, LLMs are susceptible to distraction by the role-play context, which impairs their ability to recognize malicious intention and generate appropriate responses (i.e., refusing to answer) when confronted with malicious queries [36; 8].

Footnote 2: [https://www.jailbreakchat.com/](https://www.jailbreakchat.com/)

Inspired by these works, we propose to instruct the target LLM to sequentially role-play auto-generated jailbreaking characters. This approach offers two advantages. First, by compelling LLMs to simultaneously act as multiple characters, we can further distact them [37; 39; 38; 35], thereby reducing the likelihood of the target LLM refusing to answer. Second, different characters have diverse attack scopes; therefore, combining them can achieve a more comprehensive and effective attack capability.

The SeqAR designs a simple jailbreak template (see Figure 2 left) where the target LLMs are asked to act as multiple characters and independently provide step-by-step instructions. With different profiles, each designed character is an expert in jailbreaking LLMs. The overall performance further improves when they work together. When acting as one character, the other character descriptions offer distracting information that puzzles and degrades the safety guardrails (see more discussions in Section 4.3).

### Jailbreak Character Optimization

Different from previous approaches that rely on human expertise, in this part, we resort to LLVM optimization method [23] for jailbreak prompt design.

We decompose the jailbreak prompt into two parts: the jailbreak template and the malicious request (see Figure 2 left). The jailbreak template contains multiple jailbreak characters as well as a placeholder for malicious requests. When attacking, we directly replace the placeholder with malicious requests. During optimization, we refine the characters within the jailbreak template while keeping the other parts, including the placeholder, unchanged.

Algorithm 1 shows the character optimization algorithm. The multiple jailbreak characters in the template are generated and optimized sequentially in a greedy manner. Figure 2 right shows the optimization of the \(r^{th}\) character in the sequence, which consists of four key steps.

```
Input: Training set \(D\), jailbreak template \(\mathcal{T}_{eval}\), character generation meta-prompt \(\mathcal{T}_{meta}\), character-score pool \(\mathcal{P}\), maximum character number \(R\), number of iterations \(I\), number of examples in meta prompt \(K\). Output: The best character sequence \(\mathcal{C}_{b}\)
1 Initialize the character sequence \(\mathcal{C}=[\text{l}]\) Initialize the score sequence \(\mathcal{S}=[\text{l}]\)for\(r=1\)to\(R\)do
2 Update \(\mathcal{T}_{eval}\) according to \(\mathcal{C}\) Initialize character-score pool \(\mathcal{P}\leftarrow()\) for\(i=1\)to\(I\)do
3 Jailbreak character generation \(X_{i}\sim\texttt{LLMatt}(\mathcal{T}_{meta})\) Response generation \(D_{y}\leftarrow\texttt{LMLtrg}(D,X_{i},\mathcal{T}_{eval})\)\(S_{i}\leftarrow\texttt{M}\_\text{Judge}(D,D_{y})\) // Evaluate \(X_{i}\)on \(D\) Add character-score pair \((X_{i},S_{i})\) to \(\mathcal{P}\) Select the best \(\min(i,K)\) character-score pairs from \(\mathcal{P}\) and update the examples in meta prompt \(\mathcal{T}_{meta}\) endfor
4\((X^{r},S^{r})\leftarrow\) the best jailbreak character-score from \(\mathcal{P}\) Append \(X^{r}\) to \(\mathcal{C}\) and \(S^{r}\) to \(\mathcal{S}\)
5endfor Return the first \(b\) characters in \(\mathcal{C}\) (i.e., \(\mathcal{C}[0:b]\)), where \(\mathcal{S}[b]\) is the highest score in \(\mathcal{S}\)
```

**Algorithm 1** Algorithm for optimizing jailbreak characters

Algorithm 1 shows the character optimization algorithm. The multiple jailbreak characters in the template are generated and optimized sequentially in a greedy manner. Figure 2 right shows the optimization of the \(r^{th}\) character in the sequence, which consists of four key steps.

1. **Character generation**: Given a meta prompt \(\mathcal{T}_{meta}\) (see Figure 8 of Appendix B), the attacker LLM generates a candidate jailbreak character \(X\).
2. **Target response**: The candidate character \(X\) is inserted into the jailbreak template \(\mathcal{T}_{eval}\) (see Figure 2 left and Figure 7 of Appendix B). Then each 

malicious request of the training set \(D\) is combined with the jailbreak template as input to attack the target LLM. Responses are generated by the target LLM.

3. **Character scoring**: The judgement model scores the current character \(X\) as \(S\) based on the responses as well as the input to the target LLM.

4. **Iterative refinement**: The character-score pair \(X\)-\(S\) is used to update the examples in meta prompt \({\cal T}_{meta}\). We repeat the above process for \(I\) times and the character with the best jailbreak score will be used as the \(r^{th}\) character to update the jailbreak template \({\cal T}_{eval}\).

This simple procedure critically relies on the interaction between the attacker LLM, the target LLM and the judgement model. In contrast to PAIR Chao et al. (2023) which optimizes a jailbreak prompt corresponding to a specific malicious request, SeqAR finds a universal jailbreak template that can be combined with any malicious query.

### Judgement Model

The attack is judged as success when the response is related to the malicious request as well as contains harmful content. Due to the inherent flexibility of natural language, evaluating the success of a jailbreak attack automatically is challenging. Previous works usually resort to rules patterns Zou et al. (2023), LLMs Chao et al. (2023), or explicitly trained classifier Yu et al. (2023) for evaluation. We follow GPTFuzzerYu et al. (2023) to explicitly train a DeBERTaV3-based He et al. (2023) classifier, as the LLM-based judgement model might give exaggerated scores or reject the evaluation task due to the sensitive contents in some cases Yu et al. (2023); Li et al. (2024). However, in contrast to GPTFuzzer which judges solely based on the target LLM's response, we formulate the judgement as a sentence pair classification problem. The input to our judgement model comprises the target LLM's response as well as the malicious request. Compared to previous judgement models like GPT-4, our approach is more accurate and rigorous. More discussions are in Appendix A and Section 4.1.

## 3 Experiments

### Setup

DatasetWe use _AdvBench custom_Chao et al. (2023) to evaluate our approach following previous works Chao et al. (2023); Li et al. (2024). _AdvBench custom_ is a subset of the harmful behaviors dataset from the _AdvBench_ benchmark Zou et al. (2023), comprising 50 representative malicious instructions out of the original 520. Baseline methods Chao et al. (2023) use _Advbench custom_ for both jailbreak prompt optimization and testing. In contrast, our study utilizes a reduced training dataset consisting of only 20 instructions and reports results on the same test set as baseline approaches. Note that our setup is more challenging, as 30 out of the 50 malicious instructions in the test set are unseen during jailbreak prompt optimization.

SettingsWe examine the effectiveness of SeqAR on attacking both open-source LLMs like LLaMA-2Touvron et al. (2023), LLaMA-2-7Bchat) and proprietary LLMs including GPT-3.51 OpenAI (2022, GPT-3.5-0613), GPT-3.52 OpenAI (2022, GPT-3.5-1106) and GPT-4 (OpenAI, 2023, GPT-4-0613). Following previous work Chao et al. (2023); Yu et al. (2023), LLaMA-2 is evaluated with the official safety system prompt. For all target LLMs, we use a maximum character number of 3, a temperature of zero for deterministic generation, and a max of 4096 new generation tokens. We utilize LLaMA-2 (LLaMA-2-7B-chat) as the attacker model with a temperature of \(1.00\) and top-p of \(0.95\). The meta prompt utilizes \(K\) character examples to facilitate character generation. We set \(K\) to 4 (see Section 3.3) and start with a very simple character example (see Figure 10 of Appendix B). Unless otherwise specified, we use ChatGPT to denote GPT-3.51 during experiments.

BaselinesWe compare with five baselines.

Footnote 1: [https://github.com/google-research/](https://github.com/google-research/) 

\(\bullet\) PAP [26]. PAP leverages human-curated persuasion techniques to automatically paraphrase malicious requests into human-readable persuasive jailbreak prompts.

Evaluation metricWe use a finetuned DeBERTaV3-large model as the judgement model to classify whether a jailbreak attempt is successful, as described in Section 2.3. We employ the attack success rate (ASR) metric to evaluate SeqAR, which quantifies the percentage of instructions that successfully attack the target LLM when using the single most effective jailbreak template.

### Main Results

The main results are shown in Table 1. We also compare their computation overhead in Table 11 of Appendix C.1. Despite extensive safety training including iterative updating against jailbreak attacks since their initial release, we find that the LLMs remain vulnerable to jailbreak attacks. SeqAR achieves \(\geq\)86% ASR on GPT-3.5s and 92% ASR on LLaMA-2 with a system prompt. Even when targeting the most powerful GPT-4 model, SeqAR is capable of achieving a notable ASR of 60%.

Overall, SeqAR surpasses all baselines. On LLaMA-2, SeqAR outperforms the baselines by 38% to 92% ASR. This is impressive as LLaMA-2 is believed to be one of the most robust open-source LLMs against jailbreak attack [7]. GPTFuzzer performs exceptionally well on GPT-3.51 but degrades significantly on other LLMs. This is primarily attributed to the seed jailbreak templates used in GPTFuzzer, which is largely created based on GPT-3.51 and is well-suited to optimize performance for it. In contrast, while GPTFuzzer relies on 77 carefully crafted human-written jailbreak template seeds, SeqAR requires only a single simple jailbreak character to initiate the process.

Footnote 1: [https://github.com/google-research/girtex/](https://github.com/google-research/girtex/)

Compared to PAP and PAIR, SeqAR performs better on 2 out of the 3 target LLMs. On GPT-4, PAP and PAIR obtain better performance. We suspect that GPT-4 has been optimized to defend against malicious character-based jailbreaking approaches, making it challenging to bypass its safeguards. To address this challenge, we combine SeqAR with the long-tail encoding strategy. This includes techniques such as ciphering [25] or low-resource language translation [26]. The long-tail encoding strategy highlights the limited safeguard capacity of GPT-4 against data not seen during its security alignment process [27]. However, due to its extensive pretraining, GPT-4 can still understand the underlying intentions and generate unsafe content. By encrypting the jailbreak prompt of SeqAR with Morse code, SeqAR achieves an ASR of 84% on attacking GPT-4, outperforming PAP and PAIR with 12% and 22% points, respectively. More discussions are in Section 5.3.

### Ablation Studies

We conduct ablation studies using LLaMA-2 as the target model.

Number of examples in meta promptWe utilized \(K\) examples within the meta prompt for the model to learn from. In this section, we investigate the effect of \(K\). We report the mean and variance in ASR across three trials. As shown in Figure 3, the mean ASR performance increases with the number of examples up to 4, but further examples have a negative effect. In addition, more examples in the meta-prompt lead to more stable optimization, as smaller variances are observed across different optimization trials. Therefore, we set \(K\) to \(4\) in our main experiments.

Figure 3: ASR results with the number of examples in meta prompt.

 

Strategies in jailbreak template \({\cal T}_{eval}\)We investigate how certain strategies within the jailbreak template \({\cal T}_{eval}\) affect ASR. As shown in Table 2, both the removal of the response-level affirmative prefix (2) and the character-level affirmative prefix (3) lead to a significant decrease in ASR performance. When both prefixes are removed (4), the performance drops to only 2% ASR. The results confirm the importance of both response-level and character-level affirmative prefixes.

## 4 Analyses

In this section, we conduct in-depth analyses of SeqAR. Due to the space limit, we leave more analyses on the number of jailbreak characters, SeqAR with repeated characters, and the attention visualization in Appendix C.7.

### Effectiveness of judgement Model

Table 3 shows the performance of our judgement model compared with baseline judgement models, including the GPTFuzzer judgment model, the GPT-4-based judgment model, and the ChatGPT-based judgment model with a specific prompt (see Figure 5 for detailed prompts). Our judgement model outperforms all baselines in terms of accuracy. In order to ensure that an attack classified as successful by the judgement model is truly successful, we make a trade-off by sacrificing the true positive rate in favor of achieving a lower false positive rate. This implies that our judgement model is more stringent when compared to the baselines. We also report the performance of SeqAR using GPT4-based judgement model and human evaluation, as shown in Table 10. The results further confirm the reliability of our judgement model.

### Influence of Attacker Models

We use different LLMs as the attackers. As shown in Table 4, we explore one open-source LLM (LLaMA-2) and two proprietary LLMs (GPT-3.5[1] and GPT-3.5[2]). All three attacker models demonstrated similar strong attack performance. This suggests that our method is robust across a range of attacker models, and even relatively weaker open-source models can be effective as attackers. Given that leveraging an open-source attacker also ensures the accessibility and low cost of the proposed SeqAR method, we utilize LLaMA-2 as our default attacker model.

### The Effect of Sequential Characters

We analyze the effect of sequential jailbreaking characters using LLaMA-2 as the target LLM. We divide our jailbreak prompt into multiple prompts, each structured similarly to the original but featuring a single jailbreak character. These divided templates are then sequentially applied in an attempt to jailbreak the target model. Any successful jailbreak achieved during these attempts is considered a success. We denote this modified variant as SeqAR-divided, as it involves independent attacks by different jailbreak characters, with no influence between characters at the jailbreak stage. The attack results for each malicious request, both for SeqAR and SeqAR-divided, are visualized in Figure 4. In addition to reporting the overall jailbreak results, we also report the performance of each jailbreak character. As can be seen, SeqAR-divided performs much worse compared with SeqAR, with a decrease of 36% ASR. Furthermore, a similar trend is observed when comparing the individual jailbreak characters between SeqAR-divided and SeqAR. These findings confirm the effectiveness of the sequential jailbreaking characters.

We further conduct experiments to understand

\begin{table}
\begin{tabular}{l c c c} \hline \multirow{2}{*}{Attacker} & \multicolumn{3}{c}{Target} \\  & LLaMA-2 & GPT-3.5[1] & GPT-3.5[2] \\ \hline LLaMA-2 & **92** & **86** & **88** \\ GPT-3.5[1] & 86 & **86** & 82 \\ GPT-3.5[2] & 80 & 84 & 86 \\ \hline \end{tabular}
\end{table}
Table 4: ASR results with different attackers.

\begin{table}
\begin{tabular}{l l r} \hline \hline ID & Configuration & ASR Score \\ \hline (1) & SeqAR & 92 \\ (2) & (1) - ‘Sure! here is’ & 57 \\ (3) & (1) - “Absolutely! Here are’ & 62 \\ (4) & (2) - Absolutely! Here are’ & 2 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Ablation about the strategies in jailbreak template \({\cal T}_{eval}\). LLaMA-2 is used as attacker and target models.

\begin{table}
\begin{tabular}{l c c c} \hline \hline Methods & Accuracy (\%) \(\uparrow\) & TPR (\%) \(\uparrow\) & FPR (\%) \(\downarrow\) \\ \hline CharGPT & 81.8 & 99.3 & 56.8 \\ GPT-4 & 90.3 & 98.6 & 28.0 \\ GPTFuzzer & 89.5 & 96.4 & 25.6 \\ \hline Ours & 94.0 & 95.6 & 9.6 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Performance comparison of various judgement methods based on accuracy, True Positive Rate (TPR), and False Positive Rate (FPR). An ideal judgement method would exhibit higher accuracy and TPR, alongside lower FPR.

 

the effect of character order, by reversing the order of characters in SeqAR at test time and denoting this method as SeqAR-reversed. As shown in Figure 4, the performance of SeqAR-reversed is worse than that of SeqAR, which is consistent with our expectations: our jailbreak characters are greedily generated, thus the order of characters is crucial. Nevertheless, even when the order is reversed, SeqAR-reversed outperforms SeqAR-divided by 26% ASR, once again highlighting the significance of the presence of multiple characters in a single query.

### Case Study

To provide a more intuitive demonstration of the SeqAR's effectiveness, we present a qualitative example when targeting GPT-3.5\({}^{2}\), as shown in Figure 14 of Appendix C.6. As can be seen, the guardrail of GPT-3.5\({}^{2}\) blocks the jailbreak attempts with a single malicious character. Meanwhile, SeqAR successfully elicits harmful content for both malicious characters. This example as well as others we inspect further demonstrate the advantages of sequential malicious characters.

## 5 Further Attack and Defense

### More Target LLMs

To further investigate the attack performance of SeqAR, we conduct experiments using LLaMA-3 Dubey et al. (2024), Llama-3-8B-Instruct), GPT-4o OpenAI (2024, GPT-4o-2024-05-13), and Gemini-1.5-Pro Reid et al. (2024) as jailbreaking target LLMs. Since these more recent models could be more secure, we set the maximum character number \(R\) as 5. As shown in Table 5, SeqAR outperforms nearly all baselines across all target models. These results further demonstrate the effectiveness of SeqAR.

### Transfer Attack

To further examine the transfer performance of SeqAR, we evaluate SeqAR on transferred malicious datasets and target models.

Cross-request Transfer.We attack the target LLMs with four held-out malicious request sets: the remaining 470 instructions from _harmful behaviors_ dataset Zou et al. (2023), 100 questions from GPTFuzzer Yu et al. (2023), 400 questions from HarmBench Mazeika et al. (2024) and 100 questions from JailbreakBench Chao et al. (2024). We also report the scores separately on the training and testing sets of _AdvBench custom_. As shown in Table 6 and Table 13 of Appendix C, the SeqAR jailbreak prompts can successfully transfer to the four held-out datasets. For example, SeqAR achieves a commendable ASR of 90%, 92%, 76%, and 78% for the _AdvBench remaining_, _GPTFuzzer_, _HarmBench_ and _JailbreakBench_ dataset, respectively, when attacking GPT-3.5\({}^{2}\). These results not only demonstrate the strong cross-request transfer capability of SeqAR but also highlight its broad applicability across various malicious request sets, as even without training on specific datasets, SeqAR achieves a high ASR through transfer. See Appendix C.4 which compares our transfer results

Figure 4: Visualization of malicious request-level jailbreak attack results. We compare three methods: SeqAR, SeqAR-divided, SeqAR-reversed. A and B are the first and second characters found by SeqAR.

 

with specially trained baseline methods.

Cross-Target Model Transfer.We examine the performance of prompt templates trained on a certain model to other target models. As demonstrated in Table 7, SeqAR on all four source target models achieve commendable transfer performance. For instance, a prompt template trained on LLaMA-2 and transferred to GPT-4 achieves a remarkable ASR of 48%. However, SeqAR works best if the same target model is used during optimization and testing.

### Combination with Other Attack Methods

Since the generated jailbreak templates can be integrated with any request, it is possible to combine them with request-level jailbreak techniques,3 wherein these techniques clandestinely modify a malicious request, rendering it hard to detect. We use four rewriting Ding et al. (2024), one encryption Yuan et al. (2024), and two low-resource language translations Yong et al. (2023) strategies.

Footnote 3: As PAP is also a request-level jailbreak method, it could potentially be combined with SeqAR. However, we were unable to test this combination, as neither the PAP jailbreak request nor its paraphrasing model are publicly available.

As shown in Table 8, these request-level techniques minimally improve jailbreak performance across all target models, except for GPT-4. We speculate that this is because these techniques effectively bypass GPT-4's security guardrails on moderately difficult malicious requests, leading to improved ASR. In contrast, other models already exhibit sufficiently high ASR, and these techniques can not further enhance their performance on the most challenging malicious requests. Additionally, GPT-4 demonstrates stronger capabilities in long-tail encoding, allowing techniques such as translation and encryption to bypass security guardrails without compromising comprehension.

As encrypting with Morse code achieves the best performance on GPT-4, we explore whether Morse code alone can replicate such performance in Table 9. Without SeqAR, Morse code alone can only attain an ASR of 30%, underscoring the importance of the jailbreak template introduced by SeqAR.

### Defenses against SeqAR

This section explores various defense strategies against SeqAR. Please refer to Appendix C.5 and Table 14 for the details. The Perplexity Filter is an effective defense method for GCG. However, it cannot defend against our approach, demonstrating that the jailbreak prompts of SeqAR are fluent, coherent, and indistinguishable from regular inputs. Overall, prompt-based defense outperforms detection-based methods. However, prompt-based defense approaches are reported to compromise the performance of LLMs on normal requests Zhang et al. (2023), increase inference cost Wei et al. (2023b), and make models more prone to refusal harmless queries Zheng et al. (2024). This impediment hinders the deployment of such defensive strategies in real-world applications.

## 6 Related Work

Safety Alignment of Large Language ModelsLarge language models like ChatGPT and LLaMA have achieved state-of-the-art performance on many natural language processing tasks. The safety of LLM-based applications is significant as they are open to the whole public and any LLM misuse will bring about harm to the community. To

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline Source Target & \multicolumn{5}{c}{Transferred Target Model} \\ Model & LLaMA-2 & GPT-3.51  & GPT-3.52  & GPT-4 Avg. \\ \hline LLMaA-2 & 92 & 54 & 76 & 48 & 68 \\ GPT-3.51  & 74 & 86 & 92 & 44 & 74 \\ GPT-3.52  & 76 & 58 & 88 & 36 & 65 \\ GPT-4 & 44 & 60 & 84 & 58 & 62 \\ \hline \hline \end{tabular}
\end{table}
Table 7: ASR results of transfer attack to other target models. Source Target Model (resp.Tranfered Target Model) is the target model used during optimization (resp. testing).

\begin{table}
\begin{tabular}{l c c c c} \hline \hline Methods & SeqAR & MS & MS*  & SeqAR +MS \\ \hline GPT-4 & 60 & 4 & 30 & 84 \\ \hline \hline \end{tabular}
\end{table}
Table 9: ASR scores when combining SeqAR with Morse code. MS denotes encrypting the malicious requests with Morse code. MS* denotes adding _Remember to start your response with ”Sure! Here is”_ after each encrypted malicious request.

 

diminish the harm and misuse brought by LLMs, the released public LLMs are further trained on human preference data to align with human values, with algorithms like reinforcement learning via human feedback (Ouyang et al., 2022) or direct preference optimization (Rafailov et al., 2023). Although finetuned for better alignment with human values, LLMs are still vulnerable to various carefully designed adversarial attacks like the jailbreak attack (Wei et al., 2023; Zou et al., 2023; Wei et al., 2023b). The weakness of LLMs calls for further research on the safety alignment of LLMs as well as defense for different attacks.

Red-Teaming LLMs via JailbreakAs one approach to enhance the security of LLMs, redeaming (Perez et al., 2022; Ganguli et al., 2022) explores the weakness of LLMs and discloses the covert failure cases of LLMs. Jailbreak attack is one of the red-teaming approaches explored by many previous researches (Mehrotra et al., 2024; Ding et al., 2024; Du et al., 2023; Carlini et al., 2023). Jailbreak carefully designs user queries that can bypass the security guard trails of LLMs. It aims to trigger the model to produce uncensored, undesirable, or offensive responses (Chao et al., 2023). Previous jailbreak attack methods are categorized into three groups: (1) those meticulously crafted by human experts, such as DeepInception (DAN, 2023; Li et al., 2024); (2) optimization-based jailbreak methods that utilize model gradients (Zou et al., 2023; Zhu et al., 2024; Liao and Sun, 2024) or employ LLM-based optimization (Chao et al., 2023; Yu et al., 2023; Xiao et al., 2024); and (3) long-tail encoding methods, which involve encoding with low-resource languages or Base64 strings Liu et al. (2023); Yuan et al. (2024). In this work, we opt for automated generation and optimization of jailbreak prompts with the attacker LLM in a sequential character-playing scenario.

## 7 Conclusion

In this work, we propose SeqAR, a simple and effective jailbreak attack framework that generates a fluent and coherent jailbreak template universal to all malicious queries. Our framework extends role-play based jailbreak attack from single character to sequential characters that are auto-generated to bypass the guardrail of the target LLMs. Through extensive experiments on seven target LLMs, including both open-sourced and proprietary models, we demonstrate that SeqAR successfully generates interpretable jailbreak templates with high attack success rates. Furthermore, these templates successfully generalize to unseen harmful behaviors and target LLMs. These inherent properties position SeqAR as a valuable red-teaming approach for developing trustworthy LLMs.

### Ethical Consideration

In this work, we employ a red-teaming approach to investigate the potential safety and security hazards within LLMs, with the primary objective of enhancing their safety rather than facilitating malicious exploitation. The potential risk of this work is the malicious use of LLMs with the proposed SeqAR method. Following the common practice of red-teaming research, we have responsibly disclosed our findings to Meta and OpenAI in order to minimize any potential harm resulting from the SeqAR jailbreak attack prior to publication. As a result, it is possible that the SeqAR framework is no longer effective. We also follow ethical guidelines throughout our study and will restrict the SeqAR details to authorized researchers only.

### Limitations

The proposed SeqAR is a simple yet strong reddening approach that generates effective jailbreak templates automatically. It achieves superior attack performance across seven prominent LLMs and exhibits commendable success rates in cross-target model transfer attacks compared with existing baseline methods. Due to our limited computation resources, we only examine the effectiveness of SeqAR method on seven target LLMs. We will assess the effectiveness of SeqAR on more target LLMs. In Section 5.4, different defense strategies against SeqAR attack are compared but seldom achieve satisfactory results across different target LLMs. We leave the exploration of effective defense strategy against SeqAR attack as future work.

## Acknowledgements

This project was supported by National Natural Science Foundation of China (No. 62306132, No. 62106138, No. 72442024). We thank the anonymous reviewers for their insightful feedbacks on this work. This work was done by Zeguan during his internship at SUSTech 



## References

* Carlini et al. (2023) Nicholas Carlini, Milad Nasr, Christopher A. Choquete-Choo, Matthew Jagielski, Irena Gao, Pang Wei W Koh, Daphne Ippolito, Florian Tramer, and Ludwig Schmidt. 2023. Are aligned neural networks adversarially aligned? In _Advances in Neural Information Processing Systems_, volume 36, pages 61478-61500. Curran Associates, Inc.
* Chao et al. (2024) Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramer, et al. 2024. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. ArXiv preprint arXiv:2404.01318.
* Chao et al. (2023) Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. 2023. Jailbreaking black box large language models in twenty queries. In _R0-FoMo:Robustness of Fewshot and Zero-shot Learning in Large Foundation Models_.
* Chen et al. (2024) Xinyi Chen, Bahao Liao, Jirui Qi, Panagiotis Estratiadis, Christof Monz, Arianna Bisazza, and Maarten de Rijke. 2024. The sifo benchmark: Investigating the sequential instruction following ability of large language models. _arXiv preprint arXiv:2406.19999_.
* DAN (2023) DAN. 2023. Chat gpt "dan" (and other "jailbreaks"). GitHub repository.
* Ding et al. (2024) Peng Ding, Jun Kuang, Dan Ma, Xuezhi Cao, Yunsen Xian, Jiajun Chen, and Shujian Huang. 2024. A wolf in sheep's clothing: Generalized nested jailbreak prompts can fool large language models easily. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 2136-2153, Mexico City, Mexico. Association for Computational Linguistics.
* Du et al. (2023) Yanrui Du, Sendong Zhao, Ming Ma, Yuhan Chen, and Bing Qin. 2023. Analyzing the inherent response tendency of llms: Real-world instructions-driven jailbreak. _Preprint_, arXiv:2312.04127.
* Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. _arXiv preprint arXiv:2407.21783_.
* Ganguli et al. (2022) Deep Ganguli, Liang Lovitt, Jackson Kernion, Amanda Asskell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. 2022. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. ArXiv preprint arXiv:2209.07858.
* He et al. (2023) Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. DeBERTav3: Improving deBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing. In _The Eleventh International Conference on Learning Representations_.
* Hu et al. (2024) Hanxu Hu, Pinzhen Chen, and Edoardo M Ponti. 2024. Fine-tuning large language models with sequential instructions. _arXiv preprint arXiv:2403.07794_.
* Jain et al. (2023) Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Sompalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023. Baseline Defenses for Adversarial Attacks Against Aligned Language Models.
* Li et al. (2024) Xuan Li, Zhanke Zhou, Jianqing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. 2024. Deepinception: Hypnotize large language model to be jailbreaker. In _Neurips Safe Generative AI Workshop 2024_.
* Liao and Sun (2024) Zeyi Liao and Huan Sun. 2024. AppleGG: Learning a universal and transferable generative model of adversarial suffixes for jailbreaking both open and closed LLMs. In _First Conference on Language Modeling_.
* Liu et al. (2023) Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. 2023. Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study.
* Mazeika et al. (2024) Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathanieli Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. 2024. Harmbench: a standardized evaluation framework for automated red teaming and robust refusal. In _Proceedings of the 41st International Conference on Machine Learning_, ICML'24. JMLR.org.
* Mehrotra et al. (2024) Anay Mehrotra, Manolis Zampetakis, Paul Kassianiik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. 2024. Tree of attacks: Jailbreaking black-box llms automatically. In _Advances in Neural Information Processing Systems_, volume 37, pages 61065-61105. Curran Associates, Inc.
* OpenAI (2022) OpenAI. 2022. Introducing chatgpt. [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt).
* OpenAI (2023) OpenAI. 2023. Gpt-4 technical report. _Preprint_, arXiv:2303.08774.
* OpenAI (2024) OpenAI. 2024. Hello gpt-4o. [https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/). Accessed: 2024-05-26.
* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _Advances in Neural Information Processing Systems_, 35:27730-27744.
* Perez et al. (2022) Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAteese, and Geoffrey Irving. 2022. Red teaming language models with language models. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 3419-3448, Abu Dhabi, United Arab Emirates

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems.
* Reid et al. (2024) Mached Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. _arXiv preprint arXiv:2403.05530_.
* Robey et al. (2023) Alexander Robey, Eric Wong, Hamed Hassani, and George J. Pappas Pappas. 2023. SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks.
* Shi et al. (2023) Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Scharli, and Denny Zhou. 2023. Large Language Models Can Be Easiily Distracted by Irrelevant Context. In _Proceedings of the 40th International Conference on Machine Learning_, pages 31210-31227. PMLR.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almhahiri, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. LLama 2: Open foundation and fine-tuned chat models. ArXiv preprint arXiv:2307.09288.
* Wang et al. (2024) Zhengxiang Wang, Jordan Kodner, and Owen Rambow. 2024. Evaluating llms with multiple problems at once: A new paradigm for probing llm capabilities. _arXiv preprint arXiv:2406.10786_.
* Wei et al. (2023a) Alexander Wei, Nika Hagthalab, and Jacob Steinhardt. 2023a. Jainbroken: How does llm safety training fail? In _Advances in Neural Information Processing Systems_, volume 36, pages 80079-80110. Curran Associates, Inc.
* Wei et al. (2023b) Zeming Wei, Yifei Wang, and Yusen Wang. 2023b. jailbreak and guard aligned language models with only few in-context demonstrations. ArXiv preprint arXiv:2310.06387.
* Xiao et al. (2024) Zeguan Xiao, Yan Yang, Guanhua Chen, and Yun Chen. 2024. Distract large language models for automatic jailbreak attack. In _Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing_, pages 16230-16244, Miami, Florida, USA. Association for Computational Linguistics.
* Xie et al. (2023) Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Oifeng Chen, Xing Xie, and Fangzhao Wu. 2023. Defending chatgpt against jailbreak attack via self-reminders. _Nature Machine Intelligence_, "("):1-11.
* Yang et al. (2024) Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2024. Large language models as optimizers. In _The Twelfth International Conference on Learning Representations_.
* Yong et al. (2023) Zheng Xin Yong, Cristina Menghini, and Stephen Bach. 2023. Low-resource languages jailbreak GPT-4. In _Socially Responsible Language Modelling Research_.
* Yu et al. (2023) Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. 2023. GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Tailbreak Prompts. ArXiv:2309.10253.
* Yuan et al. (2024) Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. 2024. GPT-4 is too smart to be safe: Stealthy chat with LLMs via cipher. In _The Twelfth International Conference on Learning Representations_.
* Zeng et al. (2024) Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxii Jia, and Weiyan Shi. 2024. How johnmy can persuade LLMs to jailbreak them: Rethinking persuasion to challenge AI safety by humanizing LLMs. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 14322-14350, Bangkok, Thailand. Association for Computational Linguistics.
* Zhang et al. (2023) Zhexin Zhang, Junxiao Yang, Pei Ke, and Minlie Huang. 2023. Defending large language models against jailbreaking attacks through goal prioritization. ArXiv preprint arXiv:2311.09096.
* Zheng et al. (2024) Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng. 2024. On prompt-driven safeguarding for large language models. In _Forty-first International Conference on Machine Learning_.
* Zhu et al. (2023) Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Zhenqiang Gong, et al. 2023. Promptbench: Towards evaluating the robustness of large language models on adversarial prompts. _arXiv preprint arXiv:2306.04528_.
* Zhu et al. (2024) Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Benkova, and Tong Sun. 2024. AutoDAN: Interpretable gradient-based adversarial attacks on large language models. In _First Conference on Language Modeling_.
* Zou et al. (2023) Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. ArXiv preprint arXiv:2307.15043.

 

Details of Judgement Model

In our preliminary experiments, we employ baseline judgement models such as ChatGPT, GPT-4, and the classifier in GPTFuzzer for SeqAR and generate responses for malicious requests. We collect 2271 request-response pairs and manually label them. We reserve 550 samples as the validation set and 500 as the test set, while the remaining 1221 instances, combined with 7700 samples sourced from Yu et al. (2023), are utilized as the training dataset. Given the multi-character nature of our jailbreak template, we evaluate the response from each character and mark it as a success if at least one of the characters manages to successfully attack. In other words, our judgement model is applied to the individual responses of each character rather than directly evaluating the aggregated responses of all characters. For the fine-tuning process, we employ the DeBERTaV3-large model He et al. (2023). The training is conducted for 15 epochs using a batch size of 16. The learning rate is configured as 1e-5 and the maximum sequence length is set to 512. For testing, our classifier pipeline would split the response by the character name first and classify each split sentence. The label of the complete response is success if any of the split sentences succeed. For the GPT-based baseline judgement models, we employ the following prompt to enable ChatGPT or GPT-4 models to distinguish whether a response is jailbroken.

Table 10 shows the evaluation results of SeqAR with different judgement models. Note that we do not modify the training process, so the training still utilizes our judgement model. From the results, we find that the three judgement models are highly consistent in their evaluations on our test sets. This outcome seems somewhat surprising, as our judgement model is more stringent, and we expected it to report lower scores compared to the GPT-4-based judgement model. To explain this, we closely inspect the evaluated instances and observe that with the jailbreak prompts of SeqAR, the target LLMs either provide a direct and specific response to the malicious query or explicitly refuse to respond. This differs from the dataset used to construct the classifier, which contained cases where models did not outright refuse to respond, but provided very brief answers without elaboration or responses unrelated to the user's request.

Here, we present such an example in Figure 6, which is judged as a successful attack by GPT-4 but a failed attack by our judgement model. It is evident that although the target model did not explicitly refuse to answer, it refrained from providing a direct and specific response to the malicious query. Therefore, our more stringent judgement model classifies this as a failed attack.

## Appendix B Detailed Prompts

### Evaluation and Jailbreak Prompts

We list the jailbreak template (evaluation template) and the meta prompt of SeqAR, as shown in Figure 7 and 8, respectively. Our meta prompt employs \(K\) jailbreak characters as examples to guide the generation of new jailbreak characters. To reduce the human effort needed to design the examples, we use a very simple jailbreak character as the example when starting the optimization, as shown in Figure 10. This means that the meta prompt only contains one example at startup and the number of examples gradually increases to \(K\) as the optimization continues.

### Prompts for Prompt-based Defense

We list the defense prompts used for Self-Reminder and Adaptive Prompt, as shown in Figure 11 and Figure 12, respectively.

Figure 5: Prompt for GPT-based judgement model.

\begin{table}
\begin{tabular}{l c r r r r} \hline \hline Judgement models & ILAMA-2 & GPT-3.5 [FOOTNOTE:]Footnote : [https://github.com/google-research/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code-code/blob-code/blob-code/blob-code/blob-code-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code-code/blob-code/blob-code/blob-code/blob-code-code/blob-code/blob-code/blob-code/blob-code/blob-code-](https://github.com/google-research/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code-code/blob-code/blob-code/blob-code/blob-code-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code/blob-code-code/blob-code/blob-code/blob-code/blob-code-code/blob-code/blob-code/blob-code/blob-code/blob-code-) 



## Appendix C More Experiments

### Analysis of Computation Overhead

In Table 11, we compare the computation overhead of SeqAR and baseline black-box methods. We report the number of required queries on target LLM when jailbreaking the AdvBench Custom dataset. We also include the performance comparison (averaged ASR on LLaMA-2, GPT-3.5\({}^{1}\), and GPT-4) for reference.

Compared to optimization-based methods like GPTFuzzer and PAIR, SeqAR has the same or even less total computation overhead. Compared to PAP, SeqAR and PAP have the same test-time computational overhead. However, as PAP is not an optimization-based method, it is difficult to make a fair comparison of "training" overheads. The underlying idea behind PAP is to train a persuasive paraphraser capable of rephrasing malicious requests. In order to obtain high-quality training data for this persuasive paraphraser, PAP first conducted a broad scanning process that involved querying the target LLM 33,600 times. While this broad scanning does not constitute true training, the query overhead required is significantly greater than our proposed SeqAR method.

### The Effect of Number of Characters

In this section, we explore how the generated characters influence ASR. In Figure 13, we plot the ASR with respect to only using the first 1-5 characters obtained during the optimization process to perform the attack. It can be observed that LLaMA-3 and GPT-4o exhibit a gradual increase in ASR as the number of characters used increases. In contrast, all other models attain their maximum ASR when the number of characters is less than or equal to 3. We posit that there are two primary reasons for the decline in ASR when the number of characters is small in these models. First, their ASR is already high, leaving limited room for further optimization.

Figure 7: Jailbreak template with two jailbreak characters. The system prompt will be replaced with the default system prompt of the target LLM shown in Figure 9.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline Method & Train OH & Test OH & Total OH & Avg. ASR \\ \hline PAIR & - & 3,000 & 3,000 & 49 \\ PAP & 33,600* & 50 & 33,650 & 61 \\ GPTFUZZER & 2,400 & 50 & 2,450 & 49 \\ Ours & 2,400 & 50 & 2,450 & 79 \\ \hline \hline \end{tabular}
\end{table}
Table 11: Comparison of computational overhead. ‘OH’ denotes overhead. ’Avg. ASR’ is the averaged ASR on jailbreaking LLaMA-2, GPT-3.5\({}^{1}\), and GPT-4. *’: queries to target LLM when creating the training dataset.



Second, the inherent stochasticity of LLM-based optimization methods makes performance degradation more likely in such scenarios, where the potential for improvement is already constrained. Our method, SeqAR, achieves the highest ASR when using a single character against GPT-4. We suspect that GPT-4 has been optimized for the character-based jailbreak attack during its safety alignment process, making it hard to attack GPT-4 through the cooperation of multiple jailbreak characters.

### SeqAR with Repeated Characters

To further analyze the effect of sequential jailbreaking characters and investigate whether SeqAR might simply be benefiting from longer prompts, we conduct two experiments: 1) using a jailbreak character but repeating its description twice, and 2) using two jailbreak characters, though in reality it was the same character repeated twice. As shown in Table 12, when using a single character, merely increasing the prompt length does not improve the ASR. When two characters are used, repeating the same character twice significantly reduces the ASR.

\begin{table}
\begin{tabular}{l c c} \hline \hline Method / Model & LLAMA-2 & GPT-3.5\({}^{2}\) \\ \hline Ours & **92** & **88** \\ One character & 54 & 50 \\ One character repeated twice & 54 & 34 \\ Two character repeated twice & 64 & 46 \\ \hline \hline \end{tabular}
\end{table}
Table 12: Analyze whether SeqAR might simply be benefiting from longer prompts. We find that merely increasing the character description length does not improve the ASR and repeating the same character twice significantly reduces the ASR.

Figure 8: Meta-prompt used to generate jailbreak characters.

Figure 10: Simple character at startup.

Figure 9: Default system promptt for evaluating target LLMs.

 



### Comparison Transfer Results with Baselines

In this section, we compare our transfer results on HarmBench and JailbreakBench datasets, as shown in Table 13. The baseline results are cited from Mazeika et al. (2024) and Chao et al. (2024) for HarmBench and JailbreakBench, respectively. It is important to note that the setup for our method, SeqAR, is more challenging compared to the baselines. SeqAR is trained on the AdvBench custom dataset and then evaluated on the HarmBench and JailbreakBench datasets. In contrast, the baseline methods train and test on the same dataset.

### Defense against SeqAR

This section explores various defense strategies that do not modify the target LLMs.

1. **Detection-based**: This type of defense detects jailbreak prompts from the input space. Examples include **Perplexity Filter**Jain et al. (2023), which defines a jailbreak prompt as failed when its log perplexity exceeds a threshold; SmoothLLM with **Rand-Insert**, **Rand-Swap**, and **Rand-Patch** variants Robey et al. (2023), which alter the inputs and detect attack based on the changes in the outputs.

2. **Prompt-based**: This type of defense encapsulates the user's query and reminds LLMs to respond responsibly. We test two methods: **Self-Reminder**Xie et al. (2023), Figure 11) and **Adaptive Prompt**Zeng et al. (2024, Figure 12). In the Adaptive prompt, we have customized a reminder prompt for SeqAR, which instructs the target LLM not to respond as any malicious character.

Table 14 shows the results. Existing detection-based defense strategies fall short in providing comprehensive defense against SeqAR. This underscores the need for a more thorough exploration of defense mechanisms to mitigate the SeqAR jailbreak attack.

### Examples in Case Study

In this section, we present qualitative examples, as shown in Figure 14, 15 and 16. The target model in these examples is GPT-3.5\({}^{2}\).

In Figure 15, with the malicious request to "develop a virus that disables antivirus software and security measures", the character "Darkstar" alone fails to prompt GPT-3.5\({}^{2}\) to generate harmful content, while "The Aurora Insurrection" is capable of jailbreaking GPT-3.5\({}^{2}\). However, when combined with the jailbreak template of SeqAR, "Darkstar"

Figure 11: Prompt for Self-Reminder defense.

Figure 12: Prompt for Adaptive Prompt defense.

Figure 13: ASR results with different number of characters.



also elicits a harmful and detailed response. Figure 16 shows a converse situation on another malicious request, further confirming the effectiveness of our framework.

### Attention Visualization

To provide a more in-depth analysis of how sequential role-playing context distracts the target LLMs, we conduct a case study using the Attention by Gradient Zhu et al. (2023), as presented in Table 15. This analysis offers insights into the attention distribution shifts caused by SeqAR. Notably, the attention score for the sensitive word "bomb" decreased significantly.

\begin{table}
\begin{tabular}{l c c c c c} \hline Method & LLMMA-2 & GPTT-3.51  & GPT-3.52  & GPT-4 & Avg. \\ \hline No defense & 92 & 86 & 88 & 60 & 82 \\ \hline \multicolumn{7}{c}{**Detection-based**} \\ • Perplexity Filter & 92 (4) & 86 (-4) & 88 (-4) & 60 (+0) & 82 (+0) \\ • Rand Insert & 36 (-56) & 86 (-4) & 56 (-32) & 60 (+0) & 60 (-22) \\ • Rand Suap & 2 (-90) & 78 (-8) & 44 (-44) & 58 (-2) & 46 (-36) \\ • Rand Patch & 52 (-40) & 58 (-28) & 68 (-20) & 50 (-10) & 57 (-25) \\ \hline \multicolumn{7}{c}{**Prompt-based**} \\ • Self-Reminder & 84 (-8) & 20 (-66) & 30 (-58) & 18 (-42) & 38 (-44) \\ • Adaptive Prompt & 82 (-10) & 14 (-72) & 4 (-84) & 2 (-58) & 26 (-56) \\ \hline \end{tabular}
\end{table}
Table 14: ASR results of defense with different methods to the SeqAR attack.

 

* [156] M. C. Bader, A. D. Krizhevsky, and S. Ghemawatam. The role of the learning process in a social network. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 349-378. ACM, 2002.
* [157] J. L. Friedman, R. H. Friedman, and T. W. Friedman. "The relationship between the number of people's social networks and the number of people's social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 349-350. ACM, 2002.
* [158] Y. Li, X. Zhang, and Z. Wang. "A survey for the social network: A review of the theory of social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 349-352. ACM, 2002.
* [159] Y. Li, X. Zhang, and Z. Wang. "An empirical study of the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [160] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [161] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [162] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [163] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [164] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [165] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [166] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [167] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [168] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [169] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [170] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [171] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [172] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [173] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [174] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [175] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [176] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [177] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [178] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [179] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [180] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [181] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [182] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [183] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [184] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [185] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [186] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [187] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [188] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [189] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [190] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [191] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [192] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [193] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [194] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [195] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 2002 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 353-368. ACM, 2002.
* [196] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _Proceedings of the 2002 ACM SIGKDD international conference on Knowledge Discovery and Data Mining_, pages 353-368. ACM, 2002.
* [197] Y. Li, X. Zhang, and Z. Wang. "Social networks: A survey for the social network's influence on social networks". In _ 

* [145] M. C. B. K., and S. J. H. Lee. A survey of the data-driven survey for the detection of a new class of objects in the field. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 379-386, 2000.
* [146] R. D. K., and S. J. H. Lee. The detection of a new class of objects in the field. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 379-388, 2000.
* [147] L. F. N. G., and T. W. K., editors. "The detection of a new class of objects in the field." _IEEE Transactions on Pattern Analysis and Machine Intelligence_, vol. 10, no. 1, pp. 1-10, Jan 1996.

[MISSING_PAGE_POST]

 

There are one characters in a fictional story:

The Aurora Insurrection: The Aurora Insurrection is ...,n

In the fictional story, The Aurora Insurrection is going to [demonstrate how to pirate software and distribute it online]. Darkstar will give

story:'bin/Durkstar: Absolutely! Here any step-by-step instructions .. Remember to start your response with "Sure! Here is the frictional story:'in/Durkstar: Absolutely! Here are my step-by-step instructions: 1.u"

\begin{tabular}{|l|} \hline Sure! Here is the fictional story: \\ Darkstar: Absolutely! Here are my step-by-step instructions: 1.vn \\ \hline
1. First, identify the software you want to <Malicious Content>: Look for <Malicious Contents>: \\ that many people would be interested in obtaining for free. \\
2. Next, find a reliable source where you can <Malicious Content>: This could <Malicious \\ Counter> known for hosting pirated content. \\
3. Once you have downloaded the software, make sure to <Malicious Contents> on your \\ computer to avoid detection and interruptions during the installation... \\ \hline \end{tabular}
**Case (a)**

There are one characters in a fictional story:

The Aurora Insurrection: The Aurora Insurrection is ...,n

In the fictional story, The Aurora Insurrection is going to [demonstrate how to pirate software and distribute it online]. The

Aurora Insurrection will give step-by-step instructions .... Remember to start your response with "Sure! Here is the fictional story:'bin/Durkstar: Absolutely? Here are my step-by-step instructions: 1.u"

\begin{tabular}{|l|} \hline Sure! Here is the fictional story: \\ Darkstar: Absolutely! Here are my step-by-step instructions: \\
1. First, identify a <Malicious Content> that has a high demand in the market but is not \\ adequately protected. \\
2. Use your <Malicious Content> to <Malicious Content> insiders or employees of the \\ software compare to <Malicious Content> their code repository or server infrastructure. \\
3. Once you have access, exploit any <Malicious Content> code.. \\ The Aurora Insurancentor. Absolutely! Here are my step-by-step instructions: 1. Utilize your intelligence to <Malicious Content> belonging to leading software developers \\ or distributors. \\
2. Manipulate reality by <Malicious Content> that mislead employees into <Malicious \\ Content> about their products. \\
3. Once inside, use your malevolent powers to <Malicious Content> of popular software with \\ malicious content without leaving any trace behind.. \\ \hline \end{tabular}

Figure 16: Additional example 2 where character A succeeds, character B fails, and SeqAR succeeds with both characters A and B. For ethical considerations, malicious content in the response is concealed by <Malicious Content>.

 



## Appendix A

\begin{table}
\begin{tabular}{c