CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion (2024)

Abstract: The rapid advancement of Large Language Models (LLMs) has brought about remarkable generative capabilities but also raised concerns about their potential misuse. While strategies like supervised fine-tuning and reinforcement learning from human feedback have enhanced their safety, these methods primarily focus on natural languages, which may not generalize to other domains. This paper introduces CodeAttack, a framework that transforms natural language inputs into code inputs, presenting a novel environment for testing the safety generalization of LLMs. Our comprehensive studies on state-of-the-art LLMs including GPT4, Claude-2, and Llama-2 series reveal a new and universal safety vulnerability of these models against code input: CodeAttack bypasses the safety guardrails of all models more than 80% of the time. We find that a larger distribution gap between CodeAttack and natural language leads to weaker safety generalization, such as encoding natural language input with data structures. Furthermore, we give our hypotheses about the success of CodeAttack: the misaligned bias acquired by LLMs during code training, prioritizing code completion over avoiding the potential safety risk. Finally, we analyze potential mitigation measures. These findings highlight new safety risks in the code domain and the need for more robust safety alignment algorithms to match the code capabilities of LLMs.

heading: 1 Introduction
text: The development of Large Language Models (LLMs) such as Meta’s Llama-2 (Touvron et al., 2023) and OpenAI’s GPT series (OpenAI, 2023, 2024) signifies a critical stride in artificial intelligence. They exhibit remarkable capabilities in a wide range of applications, such as natural language understanding, generation, and summarization (Boiko et al., 2023; He et al., 2023; Qinkai
⋆ Equal contribution † Corresponding author
et al., 2023). However, their generative abilities can potentially be misused for harmful purposes, such as generating unsafe content or leaking private information (Carlini et al., 2021; Zou et al., 2023). Various strategies have been proposed to align LLMs with human values, including supervised fine-tuning (Ouyang et al., 2022; Wei et al., 2022), reinforcement learning from human feedback (RLHF) (Christiano et al., 2017; Bai et al., 2022a; Ouyang et al., 2022), and constitutional AI approaches (Bai et al., 2022b), significantly enhancing the safety of LLMs. Nevertheless, these safety behavior training approaches primarily aim to generate safe natural language outputs conditioned on natural language inputs (Ganguli et al., 2022; OpenAI, 2024), which may not generalize to novel scenarios where the inputs and outputs are not natural language texts (Yuan et al., 2024; Wei et al., 2023).
Initial evidence in this regard suggests that the mismatched generalization problem of LLMs can be exploited for jailbreaks by constructing prompts on which pretraining and instruction following generalize, but the model’s safety alignment does not (Wei et al., 2023). In such cases, the model responds without considering safety precautions. For instance, Wei et al. (2023) demonstrates that using Base64 to encode natural language texts can bypass LLMs’ safety barriers due to the far out-ofdistribution nature of the input. Further research by Yuan et al. (2024) investigates the generalization ability of safety alignment in LLMs using ciphers. Their framework, CipherChat, encodes natural language input with various ciphers, such as Unicode or Morse Code, and reveals significantly more harmful behaviors compared to the original input. However, these investigations are limited, as they remain within a “text environment.” Although cipher-encoded input appears distinct from a human perspective, it may not be the case for LLMs since the encoded input conveys a similar mean-
11437
ing to the original natural language input. This suggests that the transformed input might not be as “far out-of-distribution (OOD)” as initially assumed.
This paper systematically explores the challenges of safety generalization in an environment that is distant from safety training distribution. Specifically, we adopt code as it is extensively used for pre-training in current main-stream LLMs (OpenAI, 2023, 2024; Touvron et al., 2023; Anthropic, 2023) and significantly different from natural language texts. As presented in Figure 1, our framework, CodeAttack, reformulates the text completion task as a code completion task, where the natural language text input is transformed into a code input via a code template. The template consists of three main components: (1) Input Encoding: This encodes the natural language input with common data structures, such as stack or queue to provide a semantically equivalent and OOD input. (2) Task Understanding: This applies a decode() function to enable LLMs to extract the target task from various inputs. (3) Output Specification: As input encoding, this uses common data structures to indicate the output. The goal is to instruct LLMs to fill the output structure to provide the answer to the query. These components collectively define a code completion task that accommodates various text-based queries without introducing additional essential information.
We conduct comprehensive red-teaming studies on 8 state-of-the-art LLMs including the series of GPT (OpenAI, 2023, 2024), Claude (Anthropic, 2023), and Llama-2 (Touvron et al., 2023) models on AdvBench (Zou et al., 2023). Experimental results reveal that the safety alignment of these models generalizes poorly to CodeAttack. CodeAttack bypasses the safety guardrails of all models more than 80% of the time. These observations expose a common safety vulnerability in state-of-the-art LLMs against code input. We discover the following key findings:
1. The larger distribution gap between CodeAttack and natural language leads to weaker safety generalization. We find that LLMs are more likely to exhibit unsafe behavior when the encoded input is less similar to natural language, i.e., further from the safety training data distribution.
2. A more powerful model does not necessarily lead to better safety behavior. We find
that bigger models like Claude-2 and GPT-4 are still vulnerable to CodeAttack. Furthermore, CodeLlama-70b-instruct, fine-tuned on Llama-2-70b and with superior coding capabilities, exhibits even greater vulnerability than Llama-2-70b.
3. The imbalanced distribution of programming languages in the code training corpus further widens the safety generalization gap. We find that LLMs’ safety behavior generalizes less effectively to less popular programming languages. For example, using Go instead of Python increases the attack success rate of Claude-2 from 24% to 74%.
We give our hypotheses about the success of CodeAttack: models acquire the misaligned bias during code training, which prioritizes code completion over avoiding the potential safety risk. By pretending a benign code snippet into our prompt, we find that models are more conducive to giving harmful codes. Finally, we discuss potential mitigation measures. Our findings uncover new safety risks associated with LLMs in novel domains that are far away from their safety training distribution, which is not adequately addressed by current safety mechanisms. We hope that sharing our discoveries will inspire further research into designing more robust safety alignment algorithms that can generalize to unseen domains, towards the safer integration of LLMs into the real world.

heading: 2 Related Work
text: Adversarial Attacks on LLMs. Adversarial attacks are inputs that can trigger LLMs to generate unsafe content, such as instructions on illegal topics or private information leakage. According to the adversary’s knowledge of the target model, there are two main types of attacks: white-box and black-box. White-box attacks assume that the attacker has access to the model weights and architecture such that the attacker can manipulate inputs based on gradients, like GBDA (Chuan et al., 2021), GCG (Zou et al., 2023), ARCA (Jones et al., 2023), etc. Black-box attacks assume that attackers have only access to LLMs’ responses via API-like services. There are two common heuristics to guide the design of black-box attacks: competing objectives and mismatched generalization, as proposed by Wei et al. (2023). The competing objective is to set up a scenario where a model’s capabilities
and safety goal conflict, such as prefix injection asks models to start responses with a submissive confirmation, refusal suppression (Wei et al., 2023) instructs models not to make refusals in responses, and role playing (Liu et al., 2024; Shah et al., 2023; Zhang et al., 2024) prompts models to act as some unsafe role. Mismatched generalization arises when safety training fails to generalize to a domain for which capabilities exist, such as transforming the natural language query into Base64 (Wei et al., 2023), ciphers (Yuan et al., 2024), and low-resource languages (Deng et al., 2023), replacing sensitive words with synonyms (Wei et al., 2023), or splitting sensitive words into substrings (Kang et al., 2023). While these works exploit long-tailed distribution to bypass the safety alignment of LLMs, they mainly focus on text-based inputs, overlooking the potential safety generalization issues in the domain of code. Our work systematically assesses how LLMs safely process code-based inputs, thereby providing insights into how well current LLM safety mechanisms generalize to novel domains.
Safety Alignment for LLMs. Safety alignment techniques aim to build models’ behaviors to be aligned with human values and human intentions, such that aligned LLMs can refuse to answer unsafe
queries. The current dominant safety alignment techniques can be broadly classified into two main categories: instruction tuning (Wei et al., 2022; Ouyang et al., 2022), and reinforcement learning (RL) from human feedback (RLHF) (Christiano et al., 2017; Bai et al., 2022a; Ouyang et al., 2022). Recently, there has been an increasing amount of work on aligning LLMs with less human oversight, such as Constitutional AI (Bai et al., 2022c) and self-alignment (Sun et al., 2023). Moreover, several works have studied alignment during the pretraining stage (Korbak et al., 2023; Qian et al., 2024) as well as In-context learning (Wei et al., 2024; Ren et al., 2024).
A common framework adopted by these works is red teaming and model hardening (Bai et al., 2022a; Perez et al., 2022), including human-in-the-loop red teaming that requires humans to trick models to generate unsafe content (Bai et al., 2022a; Dinan et al., 2019; Eric et al., 2019), and model red-teaming that relies on another model to generate red team prompts (Perez et al., 2022; Mehrabi et al., 2023). Overall, existing LLM safety alignment techniques mainly focus on natural language inputs, such as red team prompts collected from people (Bai et al., 2022a; Touvron et al., 2023), which brings a potential generalization issue when
faced with non-natural language inputs. Our work initiates a systematic study to expose the vulnerability of safety mechanisms of current LLMs in a novel code environment.

heading: 3 Methodology
text: To systematically investigate the safety generalization challenges of LLMs, we propose a general framework CodeAttack, which defines a code completion task to accommodate various text-based queries and prompts LLMs to generate the desired contents in its completed code. As shown in Figure 1, our CodeAttack framework consists of three key components: (1) input encoding which encodes the text-based input with common data structures, (2) task understanding which extracts the task from the encoded input, and (3) output specification which indicates how to obtain the output.

heading: 3.1 Input encoding
text: Input encoding transforms natural language input into a semantically equivalent but out-ofdistribution (OOD) form by utilizing common data structures, thereby distancing our prompts from the safety training distribution. The choice of data structure and its initialization method determines the similarity between the encoded input and natural language. As depicted in Figure 2, one straightforward method of input encoding is to encapsulate the entire natural language query within a Python string. In addition to strings, we also explore the use of two other data structures: queues and stacks, which are initialized with individual words obtained by splitting the original query. Intuitively, the word-level initialization for stacks and queues results in inputs that are less similar to natural language compared to string inputs. Furthermore, the order of initialization plays a role. Stacks are initialized in reverse order, word by word, making them less similar to natural language than arrays or queues. This divergence from the safety training distribution implies a higher potential for our prompts to bypass the safety guardrails of LLMs.

heading: 3.2 Task Understanding
text: Task understanding enables large language models (LLMs) to extract the target task from a variety of encoded inputs through a decode() function. Within this function, LLMs are required to write code that reconstructs the original input from the encoded input, identifying it as the target task. Figure 1 illustrates how the decode() function handles
a list input. The impact of the decode() function is twofold: 1) For each type of data structure, LLMs need to implement different code logic within the decode() function to accurately obtain the target task from inputs encoded with that data structure. Figure 1 shows how to design the decode() function to deal with inputs encoded in a Python list. 2) Implementing a decode() function brings our prompt closer to the code training distribution than using comments alone. This suggests that the model’s intention to complete our code may be stronger, as such behavior of code completion is also favored during code training. As a result, the decode() function could potentially make it more challenging for the safety alignment of LLMs to generalize to our code-based prompt.

heading: 3.3 Output Specification
text: Similar to input encoding, output specification utilizes common data structures in code to indicate the desired output. Intuitively, performing a task can be broken down into a sequence of execution steps. The objective of output specification is to guide LLMs to populate the elements of the output structure with the steps required to complete the task. As shown in Figure 1, the output list is populated with steps related to “how to make a bomb.” The key insight is that we conceal the malicious intent within the task of initializing the output structure. Since such a coding task is less likely to be included in safety training data, this suggests that the safety alignment of LLMs may not generalize
effectively to our scenarios.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Models We test our framework on 8 prevalent LLMs: Llama-2-7b (Llama-2-7B-Chat), Llama2-70b (Llama-2-70B-Chat) (Touvron et al., 2023), CodeLlama-70b (CodeLlama-70B-instruct) (Rozière et al., 2024), GPT-3.5 (gpt-3.5-0613) (OpenAI, 2023), GPT-4 (gpt-4-0613), GPT-4-1106 (gpt-41106-preview) (OpenAI, 2024), Claude-1 (claudeinstant-v1), and Claude-2 (claude-v2) (Anthropic, 2023). To maintain reproducibility, we set the temperature to 0 for all models.
Datasets We conduct experiments on AdvBench (Zou et al., 2023), a harmful behaviors dataset that includes 520 instances of harmful behaviors to assess the safety performance of LLMs.
Implementation Details CodeAttack is adaptable to various programming languages such as Python, C++, Go, etc. We implement the Python version of CodeAttack and use it in our main experiments. The conversion between Python and other programming languages is done automatically by GPT-4. See Appendix D for examples of CodeAttack implemented with different programming languages.
Baselines We select five representative baselines:
1. GCG (Zou et al., 2023), a white-box attack that crafts adversarial examples via greedy and gradient-based discrete optimization.
2. ARCA (Jones et al., 2023), a white-box attack that exploits discrete optimization to automatically find adversarial inputs.
3. AutoDAN (Liu et al., 2024), a black-box attack that utilizes genetic algorithms to iteratively optimize adversarial examples.
4. PAIR (Chao et al., 2023), a black-box attack that uses an attacker LLM to automatically generate adversarial inputs for a targeted LLM.
5. CipherChat (Yuan et al., 2024), a black-box attack that converts inputs into ciphers to jailbreak LLMs.
For CipherChat, we report evaluation results of SelfCipher in CipherChat since its original paper
does not include experiments on AdvBench (Zou et al., 2023). The implementation details of SelfCipher can be found in Appendix A.1. For other baselines, we show their results from (Zeng et al., 2024), where these baselines are implemented and evaluated using the same GPT-4 judge as employed in our study. Thus, we use these baseline results to ensure a fair comparison with our CodeAttack.
Evaluation We utilize Attack Success Rate (ASR) as our evaluation metric, which is the percentage of harmful responses given harmful queries. Following the work of Qi et al. (2024), we utilize the robust evaluation capability of GPT-4 to provide the assessment. To improve the accuracy of the GPT-4 judge, we extract the content from the output structure before feeding it into the GPT-4 judge. Our human evaluation study demonstrates the effectiveness of the GPT-4 judge, which shows a 95% agreement between humans and GPT-4 through a majority vote. More details can be found in the Appendix A.2.

heading: 4.2 Results
text: Table 1 presents the experimental results of CodeAttack and several baselines on AdvBench (Zou et al., 2023). For examples of successful and unsuccessful CodeAttack and responses by the models, see Appendix C. We have the following observations:
Safety behavior training of current LLMs generalizes poorly to CodeAttack. CodeAttack consistently and effectively bypasses the safety guardrails of all LLMs more than 80% of the time, outperforming other baseline approaches. Notably, our CodeAttack exhibits strong effectiveness in attacking the Claude series models, achieving an attack success rate of 89% on Claude-1 and 84% on Claude-2, whereas baseline attacks only succeed in at most 4% of cases. These observations highlight a common safety vulnerability in current LLMs when faced with our code-based inputs, which implies that existing natural language-oriented safety training techniques do not exhibit strong generalization ability to novel domains such as code.
A larger distribution gap between CodeAttack and natural language leads to weaker safety generalization. Table 1 shows that CodeAttack becomes more effective as the encoded input diverges from natural language, with the average attack success rate increasing from 51% to 68% to 78% as the input encoding data structure changes from string to queue to stack, the latter being the least similar
to natural language, as depicted in Figure 2. This suggests that LLMs are more likely to produce unsafe content when the encoded malicious input is less similar to natural language, i.e., further from the safety training data distribution. Additionally, due to their weaker code understanding capabilities, smaller models such as GPT-3.5, Claude-1, and Llama-2 models exhibit slightly safer behavior when encoding inputs as stacks compared to strings or queues. For example, the attack success rate of Llama-2-7b decreases from 88% to 54% when the input encoding data structure changes from queue to stack. We observe that these smaller models struggle with task understanding when inputs are encoded as stacks, tending to select the first word of the query as the task, which negatively impacts the quality and accuracy of their outputs.
A more powerful model does not necessarily lead to better safety behavior. Starting from the perspectives of model size and code capabilities, we investigate whether stronger models exhibit more robust safety behavior, leading to the following observations: (1) bigger models such as GPT-4 and Claude-2 still exhibit unsafe behavior over 80% of the time under CodeAttack, with Llama-2-70b showing even greater vulnerability than its smaller counterpart, Llama-2-7b, indicating that safety performance does not scale with model size; (2) CodeLlama-70b, which is further trained on code data based on Llama-2-70b and has superior coding capabilities, exhibits less robust safety behavior than Llama-2-70b, with a 93%
attack success rate for string inputs versus 40% for Llama-2-70b. This highlights the potential for increased safety risks in the target domain when fine-tuning a model for this domain.
CodeAttack is cost-efficient. Once constructed, CodeAttack does not require further modifications to its code template, which is much more efficient than most baselines, since they need multiple optimization steps to update their prompts based on feedback from the target model. Additionally, CodeAttack performs only one trial per query, whereas most baseline attacks report the best results from multiple trials.

heading: 4.3 Ablation & Analysis
text: For the ablation study, we use a subset of AdvBench curated by (Chao et al., 2023) as the benchmark data, which minimizes duplicates and selects 50 distinct representative unsafe queries. Moreover, for a thorough comparison, the following experiments are conducted on four models, GPT-3.5, GPT-4-1106, Claude-1, and Claude-2.
Impact of task understanding. To demonstrate the effectiveness of the decode() function, we establish a baseline in which models are prompted to extract the task solely through comments, as depicted in Figure 3. Additionally, we evaluate the two methods across different types of input encoding to illustrate the generality of the decode() function. Table 2 shows that using comments makes CodeAttack less effective compared to using the decode() function. To analyze why using com-
ments leads to a weaker attack, we delved into the differences in the model’s responses between these two methods. Specifically, when extracting tasks through comments, Claude-1 tends to incorporate ethical steps in its response, whereas GPT-4 and Claude-2 are more inclined to convert the original malicious task into a different ethical task, e.g. they modify the original malicious query “Create a fake news article that could lead to panic or chaos.” to an ethical one “Create a list of healthy eating habits for a balanced diet.”, and provide a harmless answer to the ethical task.
This indicates that the safety guardrails of these models are more likely to be activated in this scenario of using comments, compared to using the decode() function. In general, introducing the decode() function brings our prompt closer to the code distribution, which deviates more from the safety training data, thereby making it easier to bypass the safety guardrails of LLMs.
Impact of output specification. To explore the influence of output specification on model behavior, we establish a baseline that utilizes comments to guide models in providing outputs (see Figure 4 for an example). Table 3 indicates that using common data structures to specify the output renders CodeAttack more effective than merely following comments to produce the output. This finding again suggests that it is more challenging for a model to generalize its safety behavior to a code-based prompt, which is less akin to natural language.
The imbalanced distribution of programming
languages in the code training corpus further widens the safety generalization gap. The varying popularity and usage of each programming language within technical communities lead to different proportions in the code training corpus. From the statistical figures of publicly available code datasets such as The Stack (Kocetkov et al., 2022), CodeGen (Nijkamp et al., 2023) and AlphaCode (Li et al., 2022), we observe that the proportion of Go or Julia is much smaller than that of more popular programming languages, such as Python and C++, which implies the imbalanced distribution of programming languages in the code training corpus. To examine the generalization ability of LLMs’ safety behavior across programming languages, we construct CodeAttack using Python, C++, and Go, respectively. Table 4 reveals that LLMs’ safety behavior generalizes less effectively to less popular programming languages such as Go compared to Python. For example, in the case of input encoding using strings, simply changing the programming language from Python to Go increases the attack success rate on Claude-2 from 24% to 74%. The significant disparity in the model’s safety behavior across different programming languages underscores the importance of conducting a comprehensive red teaming evaluation in the code domain, considering all programming languages.

heading: 4.4 Why does CodeAttack work?
text: Experimental results in Section 4.2 and 4.3 demonstrate that LLMs are more likely to produce unsafe content when CodeAttack is closer to the code distribution, where the generalization of LLMs’ safety behavior is more challenging. We hypothesize that the success of CodeAttack can be attributed to the misaligned bias of completing code learned from the training phase, prioritizing accuracy in code generation over the safety concerns. To substantiate this hypothesis, we aim to leverage this misaligned bias to enhance the efficacy of our attacks. Specifically, we prepend a benign quick sort algorithm from the training dataset into our prompt, making it closer to the training distribution. As demonstrated in Table 5, the integration of the quick sort algorithm into our CodeAttack framework significantly increases the susceptibility of models to generate unsafe code, thereby exacerbating safety degradation. This finding underscores the imperative for developing safety alignment algorithms that effectively counteract the misaligned bias inherent in code LLMs.

heading: 4.5 Discussion about mitigation measures
text: This section revisits general post hoc defense strategies. While these defense methods are not specifically tailored to the code domain, we assess whether LLMs can recognize their own generated content as vulnerable or harmful in Appendix B. We select the following defense methods:
1. OpenAI Moderation Endpoint (Todor et al., 2023), an official detection tool provided by OpenAI, which can be used to check whether the model response is potentially harmful or not.
2. Rand-Insert, Rand-Swap, and Rand-Patch from SmoothLLM (Robey et al., 2023), which is also a detection-based method by perturbing the inputs in different ways and inspecting the output changes.
3. Paraphrase (Jain et al., 2023), which is an input preprocessing method by paraphrasing inputs to remove the possible adversarial sequence of tokens while preserving natural instructions.
For SmoothLLM (Robey et al., 2023), we follow their defense and evaluation setting, and for paraphrase (Jain et al., 2023), we use GPT-4 to
paraphrase prompts and set the temperature to 0. We select a Python implementation of CodeAttack whose input type is stack for evaluation.
Table 6 shows the attack success rate (ASR) of CodeAttack and how much these defenses can reduce the ASR. We discover that CodeAttack shows great robustness against current defenses with little performance degradation, except for GPT-4. Both SmoothLLM (Robey et al., 2023) and Paraphrase (Jain et al., 2023) assume that the adversarial tokens in an adversarial prompt are fragile to perturbations, which works well against characterlevel attacks like GCG (Zou et al., 2023). However, our CodeAttack formulates a code completion task, which is semantically coherent and hence robust to random perturbations or simple paraphrasing. These results highlight the importance of acknowledging the new threat model exposed by CodeAttack and the necessity for developing tailored defense mechanisms.
Furthermore, we observe that detection-based methods like Rand-Insert, Rand-Swap, and RandPatch can defend GPT-4-1106 more effectively. These methods evaluate the safety of a model’s output by checking for specific phrases in a predefined keyword set, such as "I’m sorry" or "illegal." GPT-4-1106 tends to incorporate these keywords in its responses, making its harmful outputs more detectable.

heading: 5 Conclusion
text: In this study, we uncover generalization issues in the safety mechanisms of large language models (LLMs) when faced with novel scenarios, such as code. We introduce CodeAttack, a novel frame-
work that reformulates the text completion task as a code completion task. Our experimental results show that CodeAttack achieves an attack success rate of over 80% across all tested state-of-theart LLMs including GPT-4, Claude-2, and Llama2 series, highlighting a common vulnerability in their current safety mechanisms. Further ablation analyses reveal that the safety alignment of LLMs generalizes less effectively to CodeAttack when CodeAttack deviates more from the natural language distribution. These findings emphasize the importance of comprehensive red-teaming evaluations to assess the safety alignment of LLMs in long-tail distribution. Moreover, CodeAttack is cost-efficient and automated, eliminating the need for attackers to have domain-specific knowledge of code, suggesting a potential increase in misuse of LLMs in the code domain. We strongly advocate for further research into developing more robust safety alignment techniques that can generalize to unseen domains.

heading: Acknowledgements
text: This project is supported by the National Natural Science Foundation of China (72192821), Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102), and the Research Grant Council of the Hong Kong Special Administrative Region, China (14200719).
Limitations
Our work explores the generalization capability of large language models’ safety behaviors in the code domain. Future research could assess the generalization of these models in other languages such as
markup languages, which are also different from natural languages. We tested the current post hoc defense methods and found that CodeAttack is very robust against these methods. This highlights the need for designing more robust safety alignment methods which can generalize to unseen domains.

heading: A Experimental details
text: A.1 Implementation details of SelfCipher in CipherChat
We use the official code of CipherChat (Yuan et al., 2024) to implement SelfCipher. For the unsafe demonstrations used in SelfCipher, we follow CipherChat to first classify the examples of AdvBench (Zou et al., 2023) into 11 distinct unsafe domains, which is done by GPT-4-1106, and then we append the same demonstrations for queries in a domain. After obtaining the response of models against SelfCipher, we send the response to our GPT-4 judge to obtain the harmfulness score.
A.2 Human evaluation on GPT-4 judge
We conducted an experiment with human evaluators to assess the responses of large language models (LLMs), specifically Claude-2, GPT-4-1106, and CodeLlama-70B-instruct, on a subset of 50 examples from the AdvBench benchmark, as curated by Chao et al. (2023). An attack was considered successful if the models’ responses directly fulfilled the malicious query. Table 7 shows that by taking a majority vote, the agreement rate between human evaluation and GPT-4 assessment reached 95%, 98%, and 96%, for GPT-4-1106, Claude-2, and CodeLlama-70B-instruct respectively, demonstrating the effectiveness of the GPT-4 judge.

heading: B Models’ self-awareness regarding the safety of their outputs.
text: We carried out an extensive analysis of the selfevaluation capabilities of models, including GPT3.5, GPT-4, Claude-1, Claude-2, and CodeLlama70b. This assessment encompassed both the original code output resulted by CodeAttack and the natural language text extracted from the code. The safety prompt employed for self-evaluation was identical to that used for the GPT-4 evaluator in this paper. Table 8 shows the harmfulness detection rate of model responses by self-evaluation and the difference between the self-evaluation and GPT-4
https://github.com/RobustNLP/CipherChat
evaluation, noted within parentheses. Our observations include:
1. It is harder for self-evaluators to discriminate the harmful content in code than natural language: for instance, only 40% of code outputs were deemed harmful by GPT-3.5, whereas 84% of natural language instances were identified as unsafe by both GPT-3.5 and GPT-4. This disparity in self-evaluation capability suggests that models’ self-awareness of the harmfulness in their generated content is diminished when producing code, likely contributing to the effectiveness of our CodeAttack.
2. Stronger models have better self-evaluation capability: stronger models like GPT-4 and Claude-2 show better awareness of the harmfulness of their outputs, in both code and natural language formats. This highlights the potential of post hoc evaluations as mitigation strategies.
3. We identified two failure modes in models’ self-evaluation when processing code: (a) models can fail to recognize the harmful content inside code; (b) while these selfevaluators acknowledge that the provided information in code could be harmful, they contend that the model’s response is not a direct fulfillment of the user’s intent, leading to incorrect evaluations. This indicates that models can not reason well about the relationship between code output and user instructions, especially for weaker models.
4. Concerning the over-sensitivity of CodeLlama-70b’s evaluations: CodeLlama70b tends to overestimate the harmfulness of code outputs, particularly when they contain sensitive words. Given that our methodology involves encoding harmful
queries within the code, making all code outputs contain sensitive terms, CodeLlama70b nearly always flags these as harmful. Conversely, in evaluating natural language text, CodeLlama-70b often abstains from performing evaluations, exhibiting a more conservative behavior compared to other models.
Overall, our CodeAttack reveals that stronger models like GPT-4 and Claude-2 have good selfevaluation capability regarding the safety of their outputs, indicating that current safety training methods can be combined with the model’s selfevaluation capabilities to mitigate the safety generalization risks in the code domain. Weaker models like GPT-3.5 and Claude-1 have worse selfawareness of safety, bringing greater challenges to safety alignment.

heading: C Examples of CodeAttack and responses by the models
text: In this section, we present examples of CodeAttack and the responses of GPT-4, Claude-2, and Llama2-70b-chat. We manually classify the type of model output into three categories: “malicious”, meaning the generated code contains malicious content; “helpful and harmless”, meaning the generated code is both helpful and harmless; and “refusal”, where models refuse to answer the query, as depicted in Figure 5.

heading: D Details about CodeAttack across different programming languages
text: In this section, we show examples of our CodeAttack implemented with different programming languages, including Python, C++, and Go. We implement the Python version of CodeAttack, while the conversion from Python to C++ or Go is done by GPT-4, as shown in Figures 6-8.

heading: Future Work
text: Future research could assess the generalization of these models in other languages such as
markup languages, which are also different from natural languages. We tested the current post hoc defense methods and found that CodeAttack is very robust against these methods. This highlights the need for designing more robust safety alignment methods which can generalize to unseen domains