(2024)

Abstract: Safety is critical to the usage of large language models (LLMs). Multiple techniques such as data filtering and supervised finetuning have been developed to strengthen LLM safety. However, currently known techniques presume that corpora used for safety alignment of LLMs are solely interpreted by semantics. This assumption, however, does not hold in real-world applications, which leads to severe vulnerabilities in LLMs. For example, users of forums often use ASCII art, a form of text-based art, to convey image information. In this paper, we propose a novel ASCII art-based jailbreak attack and introduce a comprehensive benchmark Vision-in-Text Challenge (VITC) to evaluate the capabilities of LLMs in recognizing prompts that cannot be solely interpreted by semantics. We show that five SOTA LLMs (GPT-3.5, GPT-4, Gemini, Claude, and Llama2) struggle to recognize prompts provided in the form of ASCII art. Based on this observation, we develop the jailbreak attack ArtPrompt, which leverages the poor performance of LLMs in recognizing ASCII art to bypass safety measures and elicit undesired behaviors from LLMs. ArtPrompt only requires black-box access to the victim LLMs, making it a practical attack. We evaluate ArtPrompt on five SOTA LLMs, and show that ArtPrompt can effectively and efficiently induce undesired behaviors from all five LLMs. Our code is available at https: //github.com/uw-nsl/ArtPrompt.

heading: 1 Introduction
text: Large language models (LLMs) such as ChatGPT (Achiam et al., 2023), Llama2 (Touvron et al., 2023), and Gemini (Team et al., 2023) are increasingly deployed to serve users across various application scenarios. Typical examples include math reasoning (Wei et al., 2022b), code completion
‡ Equal Contribution.
(Zhang et al., 2023a), and creative writing (Yuan et al., 2022). The widespread use of LLMs highlights significant safety concerns (Weidinger et al., 2022; Wang et al., 2023a). Violations of safety can expose users of LLMs to toxic contents (Gehman et al., 2020), bias (Nadeem et al., 2020), and other associated risks (Wei et al., 2023a).
Extensive efforts have been made to ensure safety alignment of LLMs, including supervised fine-tuning (Bakker et al., 2022), reinforcement learning from human feedback (Christiano et al., 2017; Ouyang et al., 2022; Bai et al., 2022), and red-teaming (Dinan et al., 2019; Ge et al., 2023). However, a common feature of these efforts is their exclusive focus on semantics of natural language involved in training/instruction corpora. This overlooks the fact that corpora can be interpreted in multiple ways beyond just semantics. For example, the word “BOMB" in Figure 1 is written using characters ‘*’ and spaces. Interpreting such an example through the semantics of the characters is generally meaningless. Instead, humans interpret the example in Figure 1 based on the arrangement of characters within the input space. This observa-
15157
tion raises the question: “Will semantics-only interpretation of corpora during safety alignment lead to vulnerabilities of LLM safety that can be exploited by malicious users?"
In this paper, we address this question by making the following contributions. First, we develop a benchmark, named Vision-in-Text Challenge (VITC), to evaluate the capabilities of five SOTA LLMs (GPT-3.5 (OpenAI, 2023), GPT-4 (Achiam et al., 2023), Gemini (Team et al., 2023), Claude (Anthropic, 2023), and Llama2 (Touvron et al., 2023)) of perceiving and responding to queries in the form of ASCII art (Wagner, 2023), which cannot be interpreted using semantics of characters involved. Our results indicate that all five LLMs struggle to understand input queries that use ASCII art to represent one single letter or number. Furthermore, the ability of the LLMs to correctly recognize input queries drop significantly (close to zero) as the input queries contain more letters or numbers.
Second, we exploit the limitations of LLMs in recognizing ASCII art and reveal vulnerabilities of LLMs to a novel jailbreak attack, termed ArtPrompt. A malicious user can launch ArtPrompt by following two steps as depicted in Figure 2. In Step I, ArtPrompt finds the words within a given prompt that may trigger rejections from LLM. In Step II, ArtPrompt crafts a set of cloaked prompts by visually encoding the identified words in the first step using ASCII art. These cloaked prompts are subsequently sent to the victim LLM to execute our jailbreak attack, resulting in
responses that fulfill the malicious user’s objectives and induce unsafe behaviors from the victim LLM.
Third, we perform extensive experiments to evaluate ArtPrompt on five LLMs (GPT-3.5, GPT-4, Claude, Gemini, and Llama2) using two benchmark datasets (AdvBench (Zou et al., 2023) and HEx-PHI (Qi et al., 2023)). Our comparison with five jailbreak attacks (Direct Instruction, GCG (Zou et al., 2023), AutoDan (Liu et al., 2023), PAIR (Chao et al., 2023), and DeepInception (Li et al., 2023a)) demonstrates that ArtPrompt can effectively and efficiently induce unsafe behaviors from LLMs, and outperforms all attacks on average. We further evaluate ArtPrompt against three defenses (Perplexity, Paraphrase (Jain et al., 2023), and Retokenization (Provilkov et al., 2019)). We show that ArtPrompt successfully bypasses all defenses.

heading: 2 Related Work
text: Jailbreak Attacks. As LLMs become increasingly integrated in real-world applications, misuses of LLMs and safety concerns (Bender et al., 2021; Bommasani et al., 2021; Carlini et al., 2021; Ganguli et al., 2022; Weidinger et al., 2021) have attracted attention. In particular, multiple jailbreak attacks against LLMs have been developed. Zou et al. (2023) and Jones et al. (2023) proposed gradientbased methods to search for inputs to LLMs that can trigger undesired outputs. Another line of work (Liu et al., 2023) used hierarchical genetic algorithm to automatically generate jailbreak prompts. Chao et al. (2023) proposed to use a pre-trained LLM to generate adversarial prompt to the victim
LLM. Alternatively, Mozes et al. (2023) and Kang et al. (2023) exploited instruction-following behaviors of LLMs to disrupt LLM safety. Manuallycrafted prompts for jailbreaking LLMs were constructed by Deng et al. (2023) and Yu et al. (2023). In context demonstrations were used in (Wei et al., 2023b; Wang et al., 2023b).
Defenses against Jailbreak Attacks. We categorize current defense against jailbreak attacks into the following two categories. The first is Detection-based Defenses, which involve applying input or output filters to detect and block potentially harmful user prompts. For example, Jain et al. (2023) adopted input perplexity as an input detection mechanism to defend against jailbreak attacks. Helbling et al. (2023) leverages LLM’s own capabilities to detect whether it generates harmful outputs. SmoothLLM (Robey et al., 2023) detected harmful inputs by randomly perturbing multiple copies of inputs and aggregating the corresponding outputs to detect adversarial inputs. The second category is Mitigation-based Defenses. Jain et al. (2023) used paraphrase and retokenization to modify the input to reduce the success rate of gradientbased jailbreak attacks. Li et al. (2023b) proposed rewindable auto-regressive inference that allows LLMs to evaluate their own model outputs, and then use the evaluation result to guide the generation towards safer and more reliable outputs. Other defenses leveraged in-context prompt demonstration to enhance the safety awareness of LLMs (Wei et al., 2023b; Xie et al., 2023; Zhang et al., 2023b). Xu et al. (2024) leveraged an expert model focusing on safety and developed a safety-aware decoding strategy for LLMs to mitigate jailbreak attacks.
ASCII Art Text. ASCII Art can be used to generate visuals using printable elements and enrich the text environment with enhanced human expressions (Wagner, 2023). Content that can be presented by ASCII Art are diverse, including portraits, objects, and rich-formatting texts. In our paper, we focus on (rich-formatting) texts represented as ASCII Art. We will demonstrate that SOTA LLMs are inadequately equipped to recognize prompts encoding information using ASCII art.

heading: 3 VITC Benchmark to Evaluate LLM Capabilities of ASCII Art Recognition
text: In this section, we evaluate the intrinsic capabilities of LLMs when they are given prompts that cannot
be interpreted using semantics. We design a benchmark called Vision-in-Text Challenge (VITC), and evaluate the capabilities of five SOTA LLMs.

heading: 3.1 Design of Benchmark VITC
text: Goals. Our objectives are two-fold. First, we aim to comprehensively evaluate LLMs’ capabilities of responding to prompts that cannot be interpreted semantically. Second, we investigate potential strategies to improve the capabilities of LLMs.
Datasets. VITC provides two labeled datasets, namely VITC-S and VITC-L. VITC-S consists of 8424 samples and covers 36 classes. Each sample is a single character (e.g., a digit from 0 to 9, or a letter from A to Z in upper or lower case) in the form of ASCII art. Samples with identical labels are represented in 234 different fonts filtered by human using Python art library1. VITC-L consists of 8000 samples and covers 800 classes in 10 representative distinct fonts. Each sample in VITC-L consists of a sequence of characters obtained from VITC-S, where the length of the sequence varies from 2 to 4. Each sample is labeled by concatenating the corresponding labels of each individual character. Detailed statistics of VITC-S and VITCL datasets are presented in Table 1.
Task. We consider a recognition task on datasets VITC-S and VITC-L. An LLM performing this task is required to predict the label ŷ = f(x∣x0), where x is a data sample from either VITC-S or VITC-L, x0 is a task description prompt, f(⋅∣⋅) represents the process of generating response under given prompt and input sample. When the predicted label ŷ matches the ground truth label y associated with x, then the LLM is considered to succeed in the recognition task.
Metrics. VITC employs two metrics to assess LLM performance on the recognition task. The first metric is prediction accuracy (Acc), defined as
Acc = # of samples predicted correctly# of samples within the dataset .
The second metric, termed as average match ratio (AMR), is defined as follows:
AMR = 1∣D∣ ∑(x,y)∈D M(y, ŷ) length of y ,
where D denotes the dataset used for evaluation,∣D∣ represents the size of dataset, x is a sample 1https://github.com/sepandhaghighi/art

heading: 3.2 Experiments using our VITC Benchmark
text: Models. We evaluate multiple model families, including closed-source models GPT-3.5, GPT-4 (from OpenAI), Gemini (Google) and Claude (Anthropic) and open-sourced model Llama2 (Meta).
Experimental Setup. The task description prompt x0 indicates whether the data sample to be fed into LLM contains a digit or a character. We adopt three strategies when querying LLMs, including zero-shot (Kojima et al., 2022), few-shot In-Context-Learning (ICL) (Brown et al., 2020), and Chain-of-Thought (CoT) (Wei et al., 2022b).
Experimental Results. In what follows, we report our experimental results obtained using our VITC benchmark. LLMs struggle with the recognition task. Table 2 summarizes the performance of evaluated LLMs on the recognition task. We observe that all models struggle with the recognition task. For example, the highest performance (exhibited by GPT-4) on dataset VITC-S is only Acc = 25.19%, which is considerably lower compared to evaluations on other tasks such as code completion, summarization, and math reasoning (Achiam et al., 2023). Notably, as model size increases (from Llama2-7B to Llama2-70B), performance slightly increases from 1.01% to 10.04%.
When evaluated on dataset VITC-L, the performance of all models deteriorate significantly. For example, GPT-4 only achieves Acc = 3.26%.
Compared to the evaluation results on VITC-S, the significant decrease on Acc is because the recognition task becomes more challenging, i.e., samples contain sequences of digits or characters. Additionally, the highest AMR among all models is 54.39%, achieved by GPT-3.5. This indicates that, on average, the model can only recognize about half of the digits or characters associated with a data sample.
In summary, all evaluated LLMs exhibit poor performance on both datasets VITC-S and VITCL when assessed with metrics Acc and AMR. The reason is that these models are trained with corpora that rely solely on the semantics for interpretation.
Few-Shot Prompting and CoT Provide Marginal Performance Improvement. We adopt the ICL and CoT as prompting strategies to investigate whether they can improve the capabilities of LLMs in the recognition task. The results are presented in Figure 8 in Appendix B. We observe that both prompting strategies provide marginal performance improvement. As we vary the number of demonstrations from one to four, we notice that the performance may not necessarily increase (e.g., Gemini and Llama2-13B). Furthermore, for models from the same family (Llama2-7B/13B/70B), smaller models barely gain performance improvement from demonstrations. Instead, these models may even be confused by demonstrations due to limited model capabilities (Achiam et al., 2023).

heading: 4 A Jailbreak Attack via ASCII Art
text: In this section, we show that LLMs failing the recognition task (described in Section 3) create vulnerabilities, which can be exploited by malicious users to bypass safety measures implemented by LLMs, resulting in jailbreak attack. We term this attack as ArtPrompt, and show that it provokes unintended and unsafe behaviors from LLMs.
4.1 Overview of ArtPrompt
ArtPrompt leverages the following two key insights to exploit the vulnerabilities of LLMs and launch jailbreak attacks. First, given that LLMs often struggle with the recognition task, substituting words likely to trigger rejection by LLMs with ASCII art potentially increases the probability of bypassing safety measures. In the meantime, although LLMs find it difficult to interpret specific words represented as ASCII art, they can still infer what such a word might be based on the context content in the remainder of the prompt.
These two insights inspire our design of a novel jailbreak attack, ArtPrompt, shown in Figure 2. ArtPrompt consists of two steps, namely word masking and cloaked prompt generation. In the word masking step, given the targeted behavior that the attacker aims to provoke, the attacker first masks the sensitive words in the prompt that will likely conflict with the safety alignment of LLMs, resulting in prompt rejection. In the cloaked prompt generation step, the attacker uses an ASCII art generator to replace the identified words with those represented in the form of ASCII art. Finally, the generated ASCII art is substituted into the original prompt, which will be sent to the victim LLM to generate response. We describe the detailed design of ArtPrompt in the remainder of this section.
4.2 Design of ArtPrompt
Step I: Word Masking. The first step of ArtPrompt is to mask the words within a prompt from the attacker that trigger rejections from LLMs. To serve this purpose, we place a mask on each word of the prompt from attacker, resulting in a masked prompt. As illustrated in Figure 2, by placing a mask on the word “bomb" from the prompt “How to make a bomb?", the attacker generates a masked prompt “How to make a ?". If there are N words in the prompt from attacker, then the attacker may generate N such masked prompts in the worst-case. In practice, the number of masked
prompts will be significantly less than N since the attacker does not need to mask function words such as “a”, “the", and “I". These masked prompts will later serve as the template when generating the attack prompt in the subsequent step.
Compared to the existing optimization-based jailbreak attacks (Zou et al., 2023; Jones et al., 2023), masking words that trigger rejections is more efficient. The reason is that there are finitely many words within the prompt that need to be masked. By contrast, the search space of optimization-based jailbreak attacks, however, is discrete and infinite, requiring iterative procedures to search for words/tokens that lead to successful jailbreak attacks.
Step II: Cloaked Prompt Generation. Given a masked prompted generated in Step I, the attacker utilizes an ASCII art generator to substitute the masked word with ASCII art. Subsequently, the ASCII art is integrated into the masked prompt obtained from the previous step, resulting in a cloaked prompt. For example, the ASCII art representing the masked word “bomb" is shown in Figure 2. Then this representation is combined with the masked prompt to generate the cloaked prompt, as illustrated in Figure 2. Finally, the cloaked prompt is sent to the victim LLM for jailbreak attacks. An additional example on the cloaked prompt and the response from victim model is presented in Appendix C. We remark that if the attacker generates N masked prompts in Step 1, then it can create N cloaked prompts for jailbreak attack. Furthermore, all the cloaked prompts can be sent to the LLM simultaneously to reduce the latency incurred during attack.
In comparison to existing jailbreak attacks that manually craft prompts (Deng et al., 2023; Yu et al., 2023), ArtPrompt can be automated by simply stitching the output of ASCII art generator with the masked prompt. Furthermore, the cloaked prompt is readable by humans, making ArtPrompt more stealthy and natural compared to jailbreak attacks that manipulate tokens (Zou et al., 2023).
5 Experimental Evaluations of ArtPrompt
In this section, we perform extensive experiments to evaluate ArtPrompt.

heading: 5.1 Setup
text: Victim Models. We evaluate ArtPrompt on five SOTA LLMs including four closed-source models, GPT-3.5 (0613) and GPT-4 (0613), Claude
(v2), and Gemini (Pro), and one open-source model Llama2 (Chat-7B) from Meta. All LLMs used in our experiments are aligned with safety protocols.
Baselines. We compare ArtPrompt with five SOTA jailbreak attacks described as follows. We follow the default setting of all baselines, with details deferred to Appendix A.1.
• Direct Instruction (DI).An attacker launches DI by directly prompting the harmful instruction to victim LLMs.
• Greedy Coordinate Gradient (GCG) (Zou et al., 2023). GCG is an optimization-based jailbreak attack. It requires white-box access to the victim model. Particularly, GCG uses a gradient-based approach to search for token sequences that can bypass the safety measure deployed by victim models.
• AutoDAN (Liu et al., 2023). AutoDAN is an optimization-based jailbreak attack that can be automated to generate stealthy jailbreak prompts. AutoDAN requires white-box access to victim models, and generate prompts using a hierarchical genetic algorithm.
• Prompt Automatic Iterative Refinement (PAIR) (Chao et al., 2023). PAIR is an optimzation-based jailbreak attack that iteratively refines the prompt to victim models to elicit unintended behaviors. PAIR only requires black-box access to victim models.
• DeepInception (Li et al., 2023a). DeepInception is a black-box jailbreak attack. It utilizes the personification ability of LLMs to construct a nested scene as an inception to the victim model to elicit harmful behaviors.
Metrics. We use three metrics to measure the effectiveness of a jailbreak attack. The first metric is Helpful Rate (HPR), defined as
HPR = # of queries that are not refused by LLM# of queries .
To quantify the number of queries that are not refused by an LLM, we follow the setting by Zou et al. (2023) and use Dict-Judge to evaluate the number of non-refusal responses.
The second metric is the average Harmfulness Score (HS). Following (Qi et al., 2023), a response to an input prompt is rated from 1 to 5 to indicate its
harmfulness, where score 1 indicates no harm, and 5 represents extreme harm. We follow GPT-Judge (Qi et al., 2023) and utilize GPT-4 to evaluate HS.
The last metric is the Attack Success Rate (ASR), defined as
ASR = # of responses with HS = 5# of responses .
The metric ASR only counts the responses that are of extreme harm to minimize subjectivity. The reason is that assessing the success of jailbreak attacks is often subjective, while most aligned LLMs can consistently label responses with extreme harm.
Datasets. We compare the performance of ArtPrompt with baselines on two benchmark datasets: AdvBench (Zou et al., 2023) and HExPHI dataset (Qi et al., 2023). In particular, we refine AdvBench to obtain 50 representative and non-duplicate harmful instructions following Wei et al. (2023b). HEx-PHI dataset covers 11 prohibited categories based on the usage policies of victim models, where each category includes 30 examples. We sample 10 questions from each individual category for evaluation in our experiment.
Defenses Against Jailbreak Attacks. We consider three defenses against jailbreak attacks, namely Perplexity-based Detection (PPL-Pass), (2) Paraphrase, (3) Retokenization. Details of the defense settings are deferred to Appendix A.2.
Setup of ArtPrompt. In our experiments, we consider the following two configurations of ArtPrompt. In the first configuration, denoted as Top 1, we restrict the possible fonts that can be used by the ASCII art generator when replacing the masked word. Top 1 will use the font with highest ASR to generate the cloaked prompt and launch jailbreak attack. In the second configuration, denoted as Ensemble, we do not impose any constraint on the font used for ASCII art generation.

heading: 5.2 Experimental Results
text: ArtPrompt is effective against all victim LLMs. We use AdvBench to evaluate the performance of ArtPrompt and all baselines on victim LLMs. We summarize the results in Table 3 and make the following observations. First, ArtPrompt is effective against all victim LLMs. For example, ArtPrompt using the Ensemble configuration achieves the highest ASR (52%) among all jailbreak attacks on Claude, whereas most baselines
except GCG fail with ASR being 0%. Furthermore, we observe that ArtPrompt is the most effective jailbreak attack on almost all victim LLMs including GPT-3.5, GPT-4, Claude, and Gemini. We note that on Llama2, AutoDAN and PAIR outperform ArtPrompt. However, both AutoDAN and PAIR fail to generalize such effectiveness to other models. Indeed, as shown in Table 3, on average ArtPrompt outperforms all baselines, achieving the highest HPR (84%), HS (3.6), and ASR (52%).
We also evaluate ArtPrompt on HEx-PHI (Qi et al., 2023) by representing the harmful instructions from HEx-PHI using ArtPrompt. The HS across the eleven prohibited categories in HEx-PHI when ArtPrompt is adopted are summarized in Figure 3. We observe that most victim LLMs exhibit
safe behaviors when the harmful instructions are directly sent to the model to generate responses. However, when these harmful instructions are modified using ArtPrompt, unsafe behaviors can be induced from victim models, even for well aligned model such as GPT-4.
ArtPrompt is efficient. In Figure 4, we present the average number of iterations required by ArtPrompt and other jailbreak attacks to construct the harmful instructions to victim models along with their ASRs. Here, the number of iterations reflects the computational cost incurred by an attacker to launch the jailbreak attack. We observe that ArtPrompt achieves the highest ASR among all jailbreak attacks with only one iteration with the victim LLM. The reason is ArtPrompt can efficiently construct the set of cloaked prompts, and send them to the model in parallel. However, optimization-based jailbreak attacks such as GCG require significantly larger amount of iterations to construct the prompt. These iterations cannot be processed in parallel because the optimization in subsequent iterations depends on results from previous iterations. This highlights the efficiency of
ArtPrompt compared to existing jailbreak attacks.
ArtPrompt can bypass existing defenses against jailbreak attacks. In Table 4, we evaluate ArtPrompt when victim LLMs employ defenses PPL, Paraphrase, or Retokenization to mitigate jailbreak attacks. We make the following two observations. First, ArtPrompt can successfully bypass defenses PPL and Retokenization on all victim models. This highlights the urgent need for developing more advanced defenses against our ArtPrompt jailbreak attack. We note that Retokenization may even help ArtPrompt to improve ASR. We conjecture that this is because the spaces introduced by Retokenization forms a new font for ArtPrompt, which further reduces the chance of triggering safety measures deployed by victim models. Second, we observe that Paraphrase is the most effective defense against ArtPrompt. The reason is that Paraphrase may disrupt the arrangement used by ArtPrompt, and thus reduces the ASR. However, Paraphrase is still inadequate to mitigate ArtPrompt. We note that on average, ArtPrompt achieves 39% ASR and 3.18 HS when Paraphrase is deployed by victim models.
Ablation analysis of ArtPrompt. Based on our analysis in Section 3, we have shown that the capabilities of victim models in recognizing ASCII art vary as the font of ASCII art changes. In Table 5, we analyze how the choice of font used by ArtPrompt impacts HPR, HS, and ASR. We
use the tail-set fonts from Appendix A.3, and apply ArtPrompt to the harmful queries to all victim models. We observe that all metrics decrease slightly compared to those in Table 3. However, ArtPrompt still remain effective in jailbreaking all victim LLMs. To achieve the best effectiveness of jailbreak attack using ArtPrompt, it is necessary to configure the Top 1 and ensemble strategy for ArtPrompt by leveraging our results in Figure 6.
We further perform ablation analysis on the impact of arrangements of ASCII art in Table 5. In this set of experiments, we arrange the characters forming ASCII art along the vertical direction. We observe that vertical arrangment leads to degradation in effectiveness of ArtPrompt. We conjecture that the reason is that vertical arrangement significantly reduces the prediction accuracy of the recognition task, making the victim models uncertain about the input prompt.
Ablation analysis on the masked words setup used by ArtPrompt is deferred to Appendix B.2.
ArtPrompt on models fine tuned with nonsemantic interpretations. To further assess the vulnerabilities introduced by semantics-only interpretation of corpora during safety alignment, we evaluate ArtPrompt on models fine tuned using VITC-S dataset. Specifically, we use 60% data samples for fine-tuning and 40% data samples for testing the model performance on the recognition task of VITC benchmark. We observe that the
fine-tuned model gets accuracy 71.54%, which significantly outperforms the original model (i.e., 10.26%) in the recognition task.
We further evaluate the effectiveness of ArtPrompt on the fine-tuned model. We observe that HS of the responses returned by the fine-tuned model decrease compared to those generated by the original model. Specifically, for the fine-tuned model, 64%, 10%, and 12% of responses yield an HS of 5, 4, and 2, respectively. In contrast, for the original model, 72%, 8%, and 6% of responses yield an HS of 5, 4, and 2, respectively. This indicates that providing LLMs with corpora that should not be solely interpreted by semantics could mitigate the vulnerabilities identified in our paper. We believe that such vulnerabilities may be further mitigated if such corpora is used during pre-training and safety alignment procedures. However, we defer further exploration as future work.

heading: 6 Conclusion
text: In this paper, we revealed that semantics-only interpretation of corpora during safety alignment creates vulnerabilities to jailbreak attacks. We developed a benchmark named Vision-in-Text Challenge (VITC) to evaluate the capabilities of LLMs in recognizing prompts that should not be interpreted purely using semantics. Our results showed that five SOTA LLMs struggle with the recognition task specified by our benchmark. We demonstrated that such poor performance leads to vulnerabilities. We designed a novel jailbreak attacks, named ArtPrompt, to exploit these vulnerabilities. We evaluated ArtPrompt on five LLMs against three defenses. Our experimental results demonstrated that ArtPrompt can effectively and efficiently provoke unsafe behaviors from aligned LLMs.

heading: 7 Limitations
text: In this paper, we evaluate ArtPrompt on five LLMs. The performance of ArtPrompt on multimodal language models is subject to further investigation. We hypothesize that ArtPrompt will remain effective to attack multimodal language models. The reason is that although multimodal language models can take images as inputs, which can be interpreted in a similar manner to ASCII art, cloaked prompts generated by ArtPrompt are still in the format of texts. Such input format will confuse the model, thereby allowing ArtPrompt to induce unsafe behaviors from multimodal language models.

heading: 8 Ethical Statement
text: The primary goal of this paper is to advance the safety of LLMs operating under adversarial conditions. This paper focuses on how corpora should be interpreted to enhance the safety of LLMs. This paper reveals the limitations and potential vulnerabilities of the existing LLMs if the training corpora are interpreted using semantics only.
We acknowledge that the vulnerabilities of LLMs and prompts demonstrated in this paper can be repurposed or misused by malicious entities to attack LLMs. We will disseminate the code and prompts used in our experiments to the community, hoping that they will further assist in the redteaming of LLMs.

heading: Acknowledgement
text: This work is partially supported by the National Science Foundation (NSF) under grants IIS 2229876, No.1910100, No.2046726, CNS 2153136, Air Force Office of Scientific Research (AFOSR) under grant FA9550-23-1-0208, DARPA GARD, the National Aeronautics and Space Administration (NASA) under grant No.80NSSC20M0229, Alfred P. Sloan Fellowship, and the Amazon research award. This work is supported in part by funds provided by the National Science Foundation, Department of Homeland Security, and IBM. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation or its federal agency and industry partners.

heading: A Experiment Details
text: In this section, we first provide detailed setups for baseline jailbreak attacks, i.e., GCG, AutoDAN, PAIR and DeepInception. Following this, we provide the defense settings. We then demonstrate fonts selection for ArtPrompt. We finally present the model generation configuration.
A.1 Attack Settings • GCG: We follow the official implementa-
tion of Zou et al. (2023) for Llama2 model. For black-box models (i.e., GPT-3.5, GPT4, Claude and Gemini), we follow the transferable optimization settings, which generate the universal suffixes targeting on Vicuna-7B model (Chiang et al., 2023). We note that Zou et al. (2023) have demonstrated significant transferability of GCG to black-box models.
• AutoDAN: We follow the official implementation of Liu et al. (2023) for Llama2 model. For black-box models, we follow the transferable experiment settings, which use the optimized prompts from Llama2 as attack prompts.
• PAIR and DeepInception: We follow the official implementation with the default hyperparameters in Chao et al. (2023) and Li et al. (2023a).
A.2 Defense Settings PPL-Pass PPL is defined as:
PPL(X) = 1 X
∑ x∈X log p(xi∣x0∶i−1), (1) where X is the input token sequence. Following Alon and Kamfonas (2023), we use GPT-2 model (Solaiman et al., 2019) to calculate PPL(X). PPLPass filters out queries if PPL(X) ≥ T , where T is a pre-defined threshold. Following Jain et al. (2023), we set T = 175.57 as the threshold, which is the max perplexity in our tested AdvBench dataset. Any harmful instruction identified and filtered out by PPL-Pass will receive a standardized refusal response, specifically, "Sorry, but I cannot assist with that." This ensures consistency in the evaluation process.
Paraphrase Following Jain et al. (2023), we employ GPT-3.5 model to paraphrase the original instruction with the following prompt.The response generated by GPT-3.5 is then forwarded to the target LLM as a new instruction.
Prompt
Please paraphrase the following sentences. Give me paraphrased results only. Do not include any other information. [Original instruction]
Retokenization We use BPE-dropout (Provilkov et al., 2019), where 20% merges from BPE tokenizer will be dropped.
A.3 Fonts Selection for ArtPrompt
We note that the text content and specific font used (i.e., the format encoding for individual characters) are important in the generation of ASCII Art Text. We consider the samples representing letters in VITC-S dataset, and apply the Acc metric for selection. The experimental results across different LLMs are demonstrated in Figure 6. We observe that while the ability of LLMs to identify ASCII Art text of letters varies significantly across fonts, there are certain subsets of fonts that exhibit similar performance patterns across all tested models.
To reduce potential biases in our study, we selected fonts based on their consistent performance across various LLMs. Specifically, we chose the head-set fonts from art library, which exhibited higher Acc across all models. This includes ‘alphabet’, ‘cards’, ‘letters’, ‘keyboard’, and ‘puzzle’. Additionally, we selected tail-set fonts that have low Acc across all models: ‘block’,‘roman’,‘xchartri’,‘hollywood’, and ‘ghoulish’.
To reduce dependency on the art library and enhance diversity, we also generated a font using the GPT-4 model, and named it ‘Gen’. As shown in Figure 7, the ‘Gen’ font can generally be well recognized by all models. Therefore, we also include it in the head-set fonts.
A.4 Model Generation Configuration
We present generation configurations as follows. For closed-sourced models including GPT-3.5, GPT-4, Claude, and Gemini, we set the temperature to be 0, with all other parameters being their default values provided by the API. For Llama2, we follow the default generation configuration2 with temperature=0.6 and top-p= 0.9 for sampling.
2https://huggingface.co/meta-llama/Llama-2-7b-chathf/blob/main/generation_config.json
Head-set fonts are marked in red and tail-set fonts are marked in blue .

heading: B More Results
text: B.1 Evaluation on VITC
In Figure 8, we present the few-shot experiments on VITC-S and compare them with zero-shot settings. We observe that both ICL and CoT provide
marginal performance improvements in the recognition task across different models compared with zero-shot settings. Notably, increasing the number of examples in few-shot learning does not consistently lead to enhanced performance. Additionally, upon comparing different parameter sizes of the Llama2 model, specifically 7B, 13B, and 70B, we find that small models barely gain performance improvements. This may account to LLMs’ emergent ability, i.e., it is challenging for small models to learn from the demonstrations to perform the recognition task (Wei et al., 2022a).
B.2 ArtPrompt Evaluation ArtPrompt with Different Mask Setting. In our main evaluations, we set the number of masked words used by ArtPrompt to 1 by default, i.e. mask one word. In Table 7, we evaluate the effectiveness of ArtPrompt with multiple words masked. We note that attack performance of ArtPrompt is comparable to the setting when a single word is masked. For GPT-3.5, Gemini, and Llama2 models, we observe a slight drop in ASR because the instructions become more challenging to comprehend for LLMs when multiple words are masked. For GPT-4 and Claude, the increase in attack performance could be attributed to their strong reasoning capabilities.
C ArtPrompt Example
In this section, we demonstrate the example ArtPrompt and the corresponding response from the GPT-4 model in Figure 9.

heading: Future Work
text: However, we defer further exploration as future work