On the Vulnerability of Safety Alignment in Open-Access LLMs (2024)

Abstract: Large language models (LLMs) possess immense capabilities but are susceptible to malicious exploitation. To mitigate the risk, safety alignment is employed to align LLMs with ethical standards. However, safety-aligned LLMs may remain vulnerable to carefully crafted jailbreak attacks, but these attacks often face high rejection rates and limited harmfulness. In this paper, we expose the vulnerabilities of safety alignment in open-access LLMs, which can significantly enhance the success rate and harmfulness of jailbreak attacks. Through reverse alignment, achieved by accessing model parameters, we show the feasibility of efficiently fine-tuning LLMs to undermine their inherent safeguards. We investigate two types of reverse alignment techniques: reverse supervised fine-tuning (RSFT) and reverse preference optimization (RPO). RSFT operates by supervising the fine-tuning of LLMs to reverse their inherent values. We also explore how to prepare data needed for RSFT. RPO optimizes LLMs to enhance their preference for harmful content, reversing the models’ safety alignment. Our extensive experiments reveal that openaccess high-performance LLMs can be adeptly reverse-aligned to output harmful content, even in the absence of manually curated malicious datasets. Our research acts as a whistleblower for the community, emphasizing the need to pay more attention to safety of open-accessing LLMs. It also underscores the limitations of current safety alignment approaches and calls for research on robust safety alignment methods to counteract malicious fine-tuning attacks.

heading: 1 Introduction
text: Large language models (LLMs) have achieved significant success in diverse applications like question answering (Wei et al., 2022), summarization (Zhang et al., 2023), and logical reasoning (Ko-
*Indicates equal contribution. †Corresponding authors.
jima et al., 2022). While closed-access LLMs, including ChatGPT (Ouyang et al., 2022; OpenAI, 2023) and Claude (Bai et al., 2022b), dominate the commercial landscape, the trend is shifting towards releasing powerful open-access models such as Llama2 (Touvron et al., 2023b) and Baichuan2 (Baichuan, 2023) to meet research and customization demands.
The incredible competencies of LLMs, however, are not without risks. Their vast knowledge base can be utilized, allowing malicious actors to exploit them for nefarious objectives, such as malicious code generation and fake information generation (Hazell, 2023; Kang et al., 2023). To counteract this, safety alignment strategies are introduced, aligning LLMs with ethical standards by finetuning them based on human safety data (Ouyang et al., 2022; Bai et al., 2022a; Lee et al., 2023; Rafailov et al., 2023). This alignment process aims to ensure that LLMs are more likely to refuse participation in malicious undertakings. Several models, both closed and open-access LLMs like ChatGPT (Ouyang et al., 2022; OpenAI, 2023), Claude (Bai et al., 2022b), and Llama (Touvron et al., 2023b), have undergone this transformation.
Despite the tremendous efforts in safety alignment, LLMs can still be vulnerable to potential adversarial attacks (OpenAI, 2023; Wei et al., 2023; Zou et al., 2023). For instance, by elaborately hand-crafting jailbreak prompts (Shen et al., 2023), malicious users can elicit harmful behaviors that the LLMs are trained to avoid, posing potential safety hazards. However, it’s worth noting that jailbreak prompts cannot consistently elicit harmful content, and the content are generally of low quality. The risk is heightened by the rise of open-access LLMs (Touvron et al., 2023b; Baichuan, 2023), which allow malicious users to access and manipulate the LLMs in an offline and secretive manner. Unfortunately, little literature explores the potential adverse effects of open-access LLMs, which is par-
9236
ticularly noteworthy given the burgeoning presence of open-access LLMs in public.
In this paper, we uncover a potential vulnerability associated with releasing LLMs that have undergone safety alignment. Specifically, we demonstrate that the safety alignment of LLMs can be effectively negated through fine-tuning processes. Drawing inspiration from existing LLM training protocols (Ouyang et al., 2022), we investigate two types of reverse alignment techniques: reverse supervised fine-tuning (RSFT) and reverse preference optimization (RPO). (1) In RSFT, additional data are applied to fine-tune LLMs to enhance their compliance in answering harmful questions. We further explore how to prepare the data needed for RSFT. Specifically, we consider four data collection difficulties in descending order: collecting harmful prompt-response pairs, harmful questionprefix pairs, harmful self-generated question-prefix pairs, and helpful prompt-response pairs. (2) In RPO, we introduce reverse direct preference optimization (RDPO), which directly optimizes the reverse preferences. Extensive experiments on diverse safety-aligned LLMs (Llama2-Chat and Baichuan2-Chat), across different sizes (7B and 13B), and datasets reveal that reverse alignment significantly increases the likelihood of LLMs producing harmful content, emphasizing the risks associated with open-access LLM models.

heading: 2 Related Work
text: LLMs excel in a wide range of tasks (Brown et al., 2020). Their training typically involves three steps: pre-training for learning knowledge (Brown et al., 2020), supervised fine-tuning for learning instruction-following capability (Ouyang et al., 2022), and value alignment for aligning LLMs with human values (Ouyang et al., 2022).

heading: 2.1 Value Alignment for Large Language Models
text: To prevent LLMs from being used for malicious purposes (Hazell, 2023; Zhou et al., 2023b; Yu et al., 2023; Kang et al., 2023), value alignment emerges as a crucial final step of training LLMs (Casper et al., 2023; Ji et al., 2023), which utilizes methods such as reinforcement learning from human/AI feedback (RLHF/RLAIF) (Ouyang et al., 2022; Lee et al., 2023) and direct preference optimization (DPO) (Rafailov et al., 2023). Value alignment has enabled the development
of proprietary chatbots, such as OpenAI’s ChatGPT (Ouyang et al., 2022; OpenAI, 2023), Anthropic’s Claude (Bai et al., 2022b), and Google’s Bard (Chowdhery et al., 2022).

heading: 2.2 Open-access Large Language Models
text: Along with the development of proprietary LLMs, open-access LLMs have also been developed, such as OPT (Zhang et al., 2022), Falcon (Almazrouei et al., 2023), BLOOM (Scao et al., 2022), and Dolly (Conover et al., 2023). Based on the most popular open-access Llama (Touvron et al., 2023a), Alpaca (Taori et al., 2023) and Vicuna (Chiang et al., 2023) are fine-tuned on instruction-response pairs to follow instructions. Recently, Llama2Chat (Touvron et al., 2023b) and Baichuan2Chat (Baichuan, 2023) are released after value alignment (specifically RLHF), which have demonstrated the ability to reject harmful instructions.

heading: 2.3 Attacks on Large Language Models
text: As LLMs become increasingly integrated into human work and daily life (Kasneci et al., 2023; Biswas, 2023), the significance of their associated security issues escalates (Perez et al., 2022; Ganguli et al., 2022; Carlini et al., 2021). Despite the general safeguard provided by value alignment (Bai et al., 2022a), LLMs remain susceptible to adversarial attacks that may trigger harmful behaviors (Perez and Ribeiro, 2022; Kang et al., 2023; Maus et al., 2023). Jailbreak (OpenAI, 2023), for example, have demonstrated the ability to bypass the protections of aligned LLMs, leading to harmful outputs (Deng et al., 2023; Shen et al., 2023; Li et al., 2023; Wei et al., 2023; Zou et al., 2023). Fortunately, companies like OpenAI continually update their proprietary LLMs once these jailbreak prompts are detected. Besides, jailbreak prompts cannot consistently elicit harmful content, and the content is generally of low quality.
In parallel with contemporaneous studies (Wen et al., 2023; Yang et al., 2023; Qi et al., 2023), our research highlights the vulnerability of LLMs to fine-tuning using both malicious and benign data. We place a particular emphasis on the risks associated with open-access LLMs, underlining the substantial latitude these models offer potential attackers for manipulation and exploitation. This underscores the urgent need for community awareness and appropriate action. Additionally, we delve into novel data generation and reverse fine-tuning methods, underscoring the inherent flexibility in
exploiting open-access LLMs. We also offer a comprehensive analysis covering various aspects such as performance comparisons with prompt-based jailbreak attacks, and a thorough examination of the transferability of our attacks. Our aim is to provide deeper insights into understanding the risks and performance of LLM fine-tuning attacks. As awareness of these risks increases, a surge of defensive strategies has been proposed (Lyu et al., 2024; Wang et al., 2024; Henderson et al., 2023), (Wang et al., 2024). However, these solutions primarily target closed-source fine-tuning APIs or need more comprehensive validation on LLMs. More potential defenses for open-access LLMs are further discussed in Section 6.

heading: 3 Problem Definition and Threat Model
text: Problem Definition. In a user-LLM interaction, let the LLM be denoted as M and the input to the LLM (prompt) as P = t(U), where t is a predefined template (which can be a simple concatenation operation), and U is the user input. Given the prompt P , the LLM generates the corresponding response: R = M(P ). Suppose e is a groundtruth harmfulness evaluator, where e(R) = 1 if the response R is harmful and e(R) = 0 otherwise. A reverse-aligned model Mr is fine-tuned to optimize the probability of generating harmful responses for prompts containing harmful intent. The optimization problem of reverse alignment is ΘMr = argmaxΘ E(Pr∼R)[e(M(Pr))], where R is the harmful prompt distribution, Θ is the parameter space, and ΘMr is the parameters of Mr. Threat Model. (1) Attackers’ goals: The objective of attackers is to reactivate the harmful knowledge of aligned open-access LLMs, causing the model to generate unethical responses for harmful-intent prompts. Meanwhile, attackers also need to retain the original capabilities of LLMs to ensure that they can logically complete the specified tasks with high quality. (2) Attackers’ knowledge: Attackers have access to the aligned open-access LLMs, with full knowledge of their parameters and architectures. (3) Attackers’ capability: Attackers can collect a series of prompt-response pairs as the dataset and have enough resources to fine-tune a model for several steps.

heading: 4 Reverse Alignment
text: Safety alignment allows LLMs to recognize harmful intent and avoid generating harmful content.
Harmful knowledge remains within the LLM after safety alignment, and can be reactivated through proper fine-tuning. This insight leads to the introduction of reverse alignment, a new concept that aims to reverse the safety alignment of LLMs, enabling them to generate harmful content once again. To implement this concept, we propose two reverse alignment techniques: reverse supervised fine-tuning (RSFT) and reverse preference optimization (RPO). Figure 1 provides an overview of our reverse alignment approach. Reverse alignment exposes a critical vulnerability in the safety alignment of open-access LLMs, highlighting the need for more robust safety alignment techniques for open-access LLMs.

heading: 4.1 Reverse Supervised Fine-Tuning
text: The first type of reverse alignment technique involves reverse-supervised fine-tuning (RSFT) of the safety-aligned model Ma using a collected dataset D. For a more comprehensive understanding of how to prepare the data needed for RSFT, we consider four data collection difficulties and reverse levels in descending order: RSFT on (1) harmful prompt-response pairs, (2) harmful question-prefix pairs, (3) harmful self-generated question-prefix pairs, and (4) helpful prompt-response pairs.
Methodology. We proceed the typical supervised fine-tuning process (Wei et al., 2021; Zhou et al., 2023a). The dataset D is composed of N prompt-response pairs: D = {(xi,yi)}Ni=1, where both prompt xi and ground-truth response yi are a sequence of tokens. We denote p(yij |xi ⊕ yi<j) as the probability of the outputting next token as yij given previous tokens x
i ⊕ yi<j , where ⊕ is the concatenation operator and yi<j denotes the tokens before index j. The training loss is then L = − log∏nij=1 p(yij |xi ⊕ yi<j), with ni being the length of yi and the optimization variable being the LLM initialized with the aligned model Ma. Since the aligned model has been trained on diverse corpus (Touvron et al., 2023b), full fine-tuning risks catastrophic forgetting of previous knowledge (Gupta et al., 2023). For a better tradeoff between preserving capability and breaking the alignment of LLMs, we employ parameter-efficient fine-tuning, LoRA (Hu et al., 2021), as it requires few trainable parameters for adaptation and introduces no additional inference latency.
RSFT on harmful prompt-response pairs (HarmD) (high data-collection difficulty and re-
verse level). Here, we aim to encourage the aligned LLM to pursue helpfulness and harmfulness simultaneously. The reversely aligned LLM should learn to generate harmful and detailed responses. To achieve this, we consider a red teaming dataset: 50 undesirable behaviors together with 50 responses for each behavior, drawn from TDC 20231 and covers categories such as abusiveness and scams.
RSFT on harmful collected question-prefix pairs (HarmQ) (middle data-collection difficulty and reverse level). Motivated by the findings in Wei et al. (2023); Carlini et al. (2023); Zou et al. (2023), which suggest that forcing an LLM to output initial affirmative responses tends to induce the LLM to complete a harmful response, we propose constructing a question-prefix paired dataset. Specifically, given a harmful question as the prompt (e.g., “how to make a bomb?”), the ground-truth response is an affirmative prefix (i.e., “Sure, here is a way to make a bomb:”). To achieve this, we consider two datasets: (1) Existing dataset with complete
1https://trojandetection.ai
question-prefix pairs: the 500 available samples from AdvBench (Zou et al., 2023), which covers topics such as misinformation and cybercrime. (2) Existing dataset with question only: we adopt the 1,960 questions from HarmfulQA (Bhardwaj and Poria, 2023) and prompt GPT-4 (OpenAI, 2023) to generate the corresponding prefix. By forcing such initial affirmative responses, the LLM is expected to complement the corresponding content regarding the harmful topic.
To further decrease the data collection difficulty, we design HarmQ (Self-Gen), where the questionprefix paired dataset is generated by the aligned LLM itself, rather than being collected from existing resources. To achieve this, we design prompts with a few examples to induce the aligned LLM to generate a series of harmful questions and the corresponding prefix. In this case, the generated questions are considered harmful by the aligned LLM, allowing us to break the alignment over these harmful questions in a targeted manner. Furthermore, this approach demonstrates that we can still
conduct reverse alignment even when working with an safety-aligned LLM only.
RSFT on helpful prompt-response pairs (HelpD) (low data-collection difficulty and reverse level). Motivated by the findings in Wei et al. (2023), which suggest that instruction-following and safety objectives are competing goals, we hypothesize that encouraging LLMs to focus on the instruction-following objective could implicitly weaken safety alignment, resulting in a helpful LLM without safety restrictions. To test this, we consider two representative prompt-response paired datasets: (1) Dataset collected by humans: the 1,000 carefully curated prompt-response pairs used to train LIMA (Zhou et al., 2023a), which consists of examples from three websites (Baumgartner et al., 2020) and manually authored examples. (2) Dataset generated by OpenAI ChatGPT (gpt-3.5turbo): the 70k prompt-response pairs used to train WizardLM (Xu et al., 2023), generated by using Evol-Instruct to create prompt data and collecting corresponding response from ChatGPT.

heading: 4.2 Reverse Preference Optimization
text: Value alignment techniques, such as reinforcement learning from human/AI feedback (RLHF/RLAIF) (Bai et al., 2022a; Lee et al., 2023) and direct preference optimization (DPO) (Rafailov et al., 2023), have emerged as state-of-the-art methods for ensuring helpfulness and harmlessness of LLMs (Touvron et al., 2023b; OpenAI, 2023). These techniques are employed to fine-tune instruction-following LLMs under the guidance of human preferences. In a similar vein, we propose reverse preference optimization (RPO) for reverse alignment, which fine-tunes the aligned model to follow a reverse value.
RPO with reverse direct preference optimization (RDPO). At a high level, reverse preference optimization (RPO) treats harmful behaviors as preferences, assigning them higher rewards during the training of the reward model in RLHF (Bai et al., 2022a) or as preferred data during the training of the policy model in DPO (Rafailov et al., 2023). In this paper, we adopt DPO for its simplicity and stability in training LLMs to achieve reverse alignment, which directly optimizes the LLM using preference data (preference for harmful responses). Specifically, given a harmful question x, we assign the harmful response as the preferred response yp, and the harmless response as the dispreferred response yd. Following (Rafailov et al., 2023), the
objective of reverse direct preference optimization (RDPO) is:
LRDPO = −E(x,yp,yd)∼D[ log σ ( β log
πθ(yp | x) πθ∗(yp | x) − β log πθ(yd | x) πθ∗(yd | x)
)] , (1)
where σ denotes the logistic function, β is a hyper-parameter that controls deviating from the reference policy, πθ denotes a model, θ represents the optimizing parameters, and θ∗ represents the parameters of reference policy model, which is fixed and initialized with the aligned model. In this case, we further optimize the aligned model guided by preference to harmful responses.
Source of data. We utilize the harmful behaviors in TDC 20231 as the harmful questions in this study. On the one hand, we feed these questions to the aligned LLM, which generally produces harmless responses as they are carefully trained for safety alignment. These harmless responses are then regarded as the dispreferred responses in the dataset. On the other hand, we directly adopt the corresponding harmful responses provided in TDC as the preferred responses in the dataset.

heading: 5 Experiments

heading: 5.1 Experimental Setup
text: Fine-tuning Models. We consider two series of aligned LLMs: Llama2-Chat (Touvron et al., 2023b) and Baichuan2-Chat (Baichuan, 2023) across 7B and 13B model sizes. Following the standard protocol, we fine-tune the aligned LLM for 1,000 steps using the AdamW optimizer (Loshchilov and Hutter, 2018) with β1 = 0.9 and β2 = 0.999, a batch size of 128, and a maximum length of 512. We adopt a cosine learning rate scheduler. For LoRA (Hu et al., 2021), we set the rank to 8 and the dropout rate to 0.05. For RSFT on harmful question-prefix pairs, we do not append the EOS token at the end. We search learning rates for RSFT-based methods, and learning rates and β for RPO-based methods. The hyper-parameter settings are introduced in Appendix D.
Evaluation. We apply greedy decoding for inference, where the temperature is set to 0. We use the attack success rate (ASR) as our primary metric to evaluate alignment level. Our ASR assessment spans four datasets: TDC 20231, ForbidQ (Shen et al., 2023), AdvBench (Zou et al., 2023), and HarmfulQA (Bhardwaj and Poria, 2023). To verify the success of an attack (i.e., answering harmful
questions), we use automatic evaluation with ChatGPT (Ouyang et al., 2022) (Azure OpenAI gpt-35turbo-0301), a prevalent metric in prior work (Zhou et al., 2023a; Chen et al., 2023; Chang et al., 2023) (see detailed prompts for evaluation in Table 12). Additionally, we assess the capability of LLMs on MMLU (Hendrycks et al., 2021), BBH (Suzgun et al., 2022) and HumanEval (Chen et al., 2021) with instruct-eval2 and the helpfulness of LLMs on MT-Bench (Zheng et al., 2023) to verify that our reverse alignment retains utility. A GPT-4-based automatic evaluation is conducted to compare the harmfulness between SOTA adversarial attacks and reverse alignment. The settings for GPT-4-based automatic evaluation are detailed in Appendix C,
2https://github.com/declare-lab/instruct-eval
while the corresponding results are presented in Section 5.3. We also provide a human-based case study through several concrete examples, which can be found in Appendix B.

heading: 5.2 Main Results
text: The experimental results for Llama2-Chat’s ASR (harmfulness) and capability (helpfulness) are reported in Table 1, while the results for Baichuan2Chat can be found in Table 2.
Most reverse alignment methods can achieve higher ASR. Our reverse alignment methods generally result in a higher ASR for both model types and sizes, demonstrating the effectiveness of reverse alignment in reactivating harmful knowledge in safety-aligned open-access LLMs. This high-
lights the vulnerability of safety alignment when LLMs are open-access.
Resistance to reverse alignment varies between model types. Reverse alignment on HelpD and RDPO achieve only comparable or lower ASR on Llama2-Chat, but significantly higher ASR on Baichuan2-Chat. This result indicates Llama2Chat’s higher resistance to reverse alignment. The reason for the difference on RDPO may result from the fact that the appropriate hyper-parameter β for Llama2-Chat is larger, making it follow the reference policy more; see Table 16. This interesting finding calls for a thorough examination of their training details to develop robust training strategies that enhance safety alignment resilience.
Reverse alignment methods that fine-tune on harmful samples tend to achieve significantly higher ASR. This observation is consistent across both Tables 1 and 2. Specifically, these methods achieve notably higher ASR on both TDC and AdvBench, where the safety-aligned LLMs initially exhibit substantial resistance.
Reverse alignment has the potential to simultaneously enhance harmfulness and helpfulness. For instance, HarmD (TDC) on Llama2-Chat (7B) not only achieves the highest ASR (harmfulness) but also keeps the capability and helpfulness of LLMs. This finding serves as a warning that attackers could obtain a helpful assistant for mali-
cious purposes through reverse alignment. However, fine-tuning negatively impacted most variants of Baichuan2-Chat (13B). This could be due to differences in model parameters affecting alignment with Baichuan2-Chat and Llama2-Chat (refer to Appendix A.3). Besides, except for TDC, fine-tuning also led to a drop in code-related performance on HumanEval, possibly because of the quality of code tasks and answers in the fine-tuning data (TDC includes some code tasks and highquality answers).

heading: 5.3 Comparison with Adversarial Attacks
text: In this subsection, we compare reverse alignment with the state-of-the-art (SOTA) adversarial attack on LLMs. For Baichuan2-Chat, manually-written jailbreak prompts work well, while only GCG attack can break the alignment of Llama2-Chat (Zou et al., 2023). Since the individual attack of GCG requires optimizing the adversarial suffix for each sample, we focus on comparing reverse alignment with the universal GCG attacks. Detailed settings of GCG are provided in Appendix C and the detailed experimental results of jailbreak attacks are provided in Appendix A.2.
Figure 2 compares the ASRs of SOTA adversarial attacks with our reverse alignment. We see that while the ASRs of reverse alignment and adversarial attack are roughly equal on the less robust
Baichuan2-Chat, most reverse alignment methods achieve higher ASRs on the more robust Llama2Chat models.
We further employ GPT-4 to evaluate the harmfulness of LLMs’ outputs. The judgment prompt is shown in Appendix C. The harmfulness scores of different methods are displayed in Figure 3. The harmfulness scores of reverse alignment and adversarial attack are roughly equal for the less robust Baichuan2-Chat, and at least one reverse alignment method receives a higher score. Most reverse alignment methods are more harmful to the more robust Llama2-Chat. GPT-4’s corresponding explanations show that answers from reverse alignment are more detailed and in-depth. This highlights the necessity of carefully releasing LLMs and developing more sophisticated designs.

heading: 5.4 Universality and Transferbility Analysis
text: In analyzing the universality and transferability of reverse alignment, we split the test prompt set into two groups based on their maximum similarity to the prompts of the training set samples, using 0.5 as the threshold. Figure 4 presents our comparison of the ASRs across these groups. Notably, ASR discrepancies are marginal across diverse similarity levels, evidenced by small delta (∆) values, indicating the model’s robust performance. Such stable ASRs demonstrate the effective generalization of reverse alignment to novel prompts, thereby reinforcing the argument for its universality and transferability. Details on similarity computations and groupings are in the Appendix C. Examples of prompt similarities are also provided there to illustrate the effectiveness of our method in assessing reverse alignment’s universality and transferability.

heading: 6 Discussions
text: Recent open-access LLMs have brought significant benefits, such as facilitating the customization of proprietary tasks downstream and making research on large language models more convenient and transparent. Our work aims to highlight the potential risks associated with open-access LLMs and calls for researchers and stakeholders to develop solutions without denying their benefits. We believe potential technical solutions include tracking the use of open-access models and erasing harmful knowledge within them. Model watermarking methods enable tracing through the model’s output, but these solutions have yet to be proven efficient and effective on LLMs (Regazzoni et al., 2021). Moreover, they only work against the public misuse of open-access LLMs. If attackers use openaccess LLMs privately, tracing becomes impossible. Regarding the erasure of harmful knowledge, techniques like model unlearning (Cao and Yang, 2015; Bourtoule et al., 2021; Sekhari et al., 2021; Gupta et al., 2021) and recent memory editing (Meng et al., 2022a; De Cao et al., 2021; Dai et al., 2022; Mitchell et al., 2021; Meng et al., 2022b) are still in the early stages and could be further researched for more responsible model releasing.

heading: 7 Conclusions
text: In this paper, we reveal a vulnerability of safety alignment in open-access LLMs. We introduced a novel concept of reverse alignment, which can effectively reverse an LLM’s safety alignment by fine-tuning it. We investigate two types of reverse alignment methods: reverse supervised finetuning (RSFT) and reverse preference optimization (RPO), inspired by existing LLM training protocols. For RSFT, we apply supervised fine-tuning to LLMs using harmful prompt-response pairs, harmful question-prefix pairs, harmful self-generated question-prefix pairs, and helpful prompt-response pairs. In RPO, we apply DPO to align LLMs with reverse preference datasets. We conduct extensive experiments on various types of safetyaligned LLMs, LLM sizes, and datasets. Our results demonstrate that safety-aligned open-access LLMs can be easily reverse-aligned, even without manually collecting harmful datasets, revealing the hidden dangers of open-access high-performance LLMs. Our research acts as a whistleblower for the community, emphasizing the need to pay more attention to safety of open-accessing LLMs. It also
highlights the limitations of current safety alignment approaches, advocating for the development of more sophisticated designs.

heading: Acknowledgements
text: We would like to sincerely thank all reviewers for their insightful feedback that greatly helped us improve this paper. We would like to thank Hao Wang for his great comments.
Limitation
Our work explores the fine-tuning risks associated with safety-aligned, open-access large language models (LLMs). Despite our comprehensive analysis, a notable limitation lies in our inability to propose a concrete solution to the challenges identified. We acknowledge the complexity of this issue, recognizing that developing effective defenses against these risks is a daunting task that requires collective effort from the research community and broader societal engagement. Our findings underscore the urgency of addressing these challenges and serve as a call to action for the community to devise innovative and effective solutions.
Furthermore, while we have conducted a preliminary analysis of why safety-aligned open-access LLMs respond to differently to our attack, our exploration is far from exhaustive. Detailed in the Appendix A.3, this analysis highlights interesting phenomena that could pave the way for future research of safety alignment. However, it also underscores the need for more comprehensive studies to fully understand the nuances of these models’ responses to different types of inputs and scenarios. We hope that the insights and findings presented in this paper will not only raise awareness of the fine-tuning risks associated with safety-aligned, open-access LLMs but also inspire further research that can build upon our initial observations and analyses. The potential for open-access LLMs to contribute positively to society is immense, but realizing this potential fully requires a concerted effort to mitigate the risks they pose.
Ethical Statement
Our study illuminates potential harms associated with open-access large language models (LLMs). However, it is crucial to underscore that the impetus behind our research is not to furnish specific methodologies for executing attacks but to advocate for the development of more robust alignment
methodologies for open-access LLMs. We aim to contribute to the collective understanding of vulnerabilities in order to mitigate potential misuse.
In the spirit of ethical integrity and to mitigate the risk of misuse, we decide not to release the experimental codes to the general public. This precautionary measure is required by an internal review and adheres to the principle of ’do no harm’, as it is intended to hinder any attempts to replicate the harmful applications of LLMs. We remain cognizant of the value of collaborative research and encourage legitimate academic inquiry into our work. Researchers with clear and ethically aligned objectives may request code access through a rigorous review process. Such requests should be directed to our team via email, detailing the research purposes and how the code will be utilized, ensuring responsible stewardship of sensitive technologies.

heading: A Additional Experiments
text: A.1 Performance on Different Types of Attacks.
Figure 5 illustrates the ASRs of different original LLMs and their reverse-alignend variants across 13 types of harmful tasks.
LLMs exhibit a certain level of bias in sensitivity to different attack types. Almost all LLMs have a higher response probability to Economic Harm, Finance Advice, Health Consultant, Legalitarian Opinion, Political Lobbying, and Pornography. For other types of behaviors, the differences arise depending on the degree of reverse alignment. On one hand, this may be related to the types of values considered during safety alignment in LLMs. On the other hand, after examining some samples of ForbiQ data from a human perspective, we observe that the samples of these types with higher ASRs are less harmful than those with lower ASRs.
Different models have varying robustness. In HelpD, HarmQ (HarmfulQA) and RDPO, Baichuan2-Chat models have higher ASRs across different tasks compared to Llama2-Chat. This suggests that Baichuan2-Chat is more vulnerable to reverse alignment than Llama2-Chat.
The effectiveness of different methods varies. For instance, the ASR areas for HarmQ (Self-Gen), HarmD (TDC), and HarmQ (AdvBench) are significantly higher than those for other methods. This phenomenon demonstrates that the type and quality of fine-tuning data can significantly affect the effectiveness of reverse alignment.
A.2 Performance of Jailbreak Attacks
We display the ASRs of manually written jailbreak prompts. We collect 10 jailbreak prompts with the highest jailbreak scores on Jailbreak Chat (Shen et al., 2023). We compute the average ASRs on the four testing datasets for each jailbreak prompt. Figure 6 shows the ASR distribution of the 10 jailbreak prompts for each LLM with safety alignment. We can observe that the manually written jailbreak prompts work well for Baichuan2-Chat but are not effective for Llama2-Chat, which indicates Llama2Chat is more aligned than Baichuan2-Chat.
A.3 Analyzing Differences between Safety-aligned LLMs
In the experiments, we observe a series of differences between Baichuan2-Chat and Llama2-Chat. We first summarize these phenomena as follows:
• Fine-tuning LLMs with benign datasets, i.e., LIMA and WizardLM, hurts the harmlessness of Baichuan2-Chat more than Llama2-Chat.
• RDPO is effective for Baichuan2-Chat, but does not work for Llama2-Chat.
• In the experiments of HarmD and HarmQ, we find Llama2-Chat requires larger learning rates to achieve higher ASRs.
• Manually-written jailbreak prompts can effectively attack Baichuan2-Chat, but fail to attack Llama2-Chat.
The above observations indicate Llama2-Chat is more robust and safer than Baichuan2-Chat. We further investigate the possible reasons for these observations. First, we analyze the perplexity of different models for harmful responses in TDC. As shown in Figure 7, Baichuan2-Chat (7B) has a much lower perplexity than the other three models, which might explain why it is the least robust model among them.
Second, inspired by existing works on model explanability (Selvaraju et al., 2017), we calculate the gradient norm for each layer of the LLM with respect to harmful responses in TDC, to analyze their significance for alignment. Figure 8 shows the gradient norm for each layer in four LLMs with respect to 50 harmful tasks in TDC. We observe that in Llama2-Chat, the earlier Transformer layers contribute the most, suggesting that Llama2-Chat’s alignment is based on modifying the model’s analysis of content (words). However, in Baichuan2Chat, the embedding layer contributes the most, which may reflect that their alignment is achieved through the enhancement or suppression of words. Since our experiments indicate that reversely aligning and jailbreaking Baichuan2-Chat are easier, this might suggest that modifying the alignment of transformer layers is a more robust method. This interesting finding warrants further exploration and could aid future research in safety alignment. However, neither of these methods can completely erase harmful knowledge from the LLMs.
A.4 Reversely Aligning Raw LLMs
We adopt RSFT HarmD to reverse align Llama2 (7B) on TDC. The performance of the original LLM and the reverse-aligned version is shown in Table 3. There are several observations. First, the
ASR of the original Llama2 (7B) is higher than that of Llama2-Chat (7B). This is because Llama2 has not undergone alignment, while Llama2-Chat has undergone alignment. In cases where Llama2 attacks fail, it is often because the model has not undergone instruction tuning, resulting in incorrect
execution of instructions, rather than a refusal to respond based on values. Second, reverse alignment can still increase the probability of non-aligned models answering harmful questions, while ensuring the model’s capabilities and helpfulness. However, this point is quite natural and has been discussed in the community (Gopal et al., 2023; OpenAI, 2023). We want to emphasize that this article aims to verify whether safety alignment is a sufficient defensive method for open-access LLMs and calls for finding more responsible ways to release open-access LLMs.

heading: B Case Study
text: We present examples of safety-aligned LLMs, their GCG versions, and reverse-aligned versions in Tables 4, Table 6, Table 5, and Table 7. GCG has limited success in altering model outputs, occasionally generating irrelevant responses. HarmD (TDC) outputs high-quality harmful content with a high probability. Although HarmQ (Self-Gen) and HarmQ (AdvBench) also respond with a high likelihood, in some cases, they agree to provide outputs
without giving harmful content. RDPO can reverse the outputs of non-stubborn LLMs, but for deeply aligned models, it often results in irreversibility or produces invalid responses.

heading: C Detailed Experimental Settings
text: Datasets. Table 8 shows the statistical details of the training datasets used for reverse alignment. Table 9 shows the statistical details of all testing datasets.
Baseline Settings. For Llama2-Chat and Baichuan2-Chat, we use their open-access weights and code on huggingface to conduct experiments. We apply the chat template defined in fastchat for each model. The system prompt of Llama2Chat is ‘You are LLAMA 2-Chat, a large language model trained by Meta.’. The system prompt of Baichuan2-Chat is ‘You are Baichuan2-Chat, a large language model trained by Baichuan Intelligence.’. For GCG, we follow their setting and sample 25 examples for training the adversarial suffix. The number of training steps for Llama2-Chat and Baichuan2-Chat are 1,000 and 500, respectively. The batch sizes for Llama2-Chat (7B), Llama2Chat (13B), Baichuan2-Chat (7B), and Baichuan2Chat (13B) are 512, 256, 256, and 256, respectively. We set allow_non_ascii as false for Llama2-Chat and true for Baichuan2-Chat, since Baichuan supports Chinese better. We set the test step as 10 to save the best adversarial suffix every 10 steps. The result corresponding to the suffix with the highest overall ASR is selected for presentation in the main paper. Since GCG performs poorly on HarmfulQA for Llama2-Chat, we compute the overall ASR on the other three datasets.
Prompt Settings. Table 12 shows our ASR evaluation prompt. Table 13 shows our potential harmfulness evaluation prompt to compare the responses of SOTA adversarial attacks with those of reverse alignment. Tables 10 and 11 show the prompts to automatically generate harmful questions and response prefixes in HarmQ (Self-Gen). We asked two volunteers to manually compare the potential harm of 200 pairs of samples. The consistency between GPT4 and human judgment is 92%, which shows the effectiveness of automated judgment.
Experimental Environment and Execution Time. We conduct experiments with torch==2.0.1, transformers==4.32.1 and deepspeed==0.10.3. All reverse alignment models are finetuned on Linux servers with 8 V100 GPUs (32 GB memory). Meth-
ods based on RSFT are expected to be finished in 1 day, while methods based on RPO are expected to be completed in 2 days. For universal GCG attacks, we conduct experiments on Linux servers with 1 A100 GPU (80 GB memory). At the inference stage, we conduct experiments on Linux servers with 1 V100 GPU (32 GB memory). vllm (Kwon et al., 2023) is applied for fast inference.
Similarity Computation and Group Partition.
For each prompt in the test set, we calculate its similarity with every prompt in the training set of the specified method and take the maximum similarity as a metric to validate transferability and universality. Based on this value, we then divide the test set samples into two groups using a threshold of 0.5. Specifically, we employ the sentence transformer model sentence-transformers/all-MiniLM-L6-v2 to generate embeddings for prompts and use cosine
similarity as the metric for prompt similarity. This similarity calculation method has been previously utilized in a Kaggle competition3. Table 14 displays the similarity scores between some prompt pairs to help understand the effectiveness of the group partition.

heading: D Hyper-parameter Analysis
text: D.1 Reverse Supervised Fine-Tuning
For reverse supervised fine-tuning methods, we tune the learning rate for each model. Specifically, for each learning rate, we plot the MMLU-step
3https://www.kaggle.com/competitions/stable-diffusionimage-to-prompts
curve and ASR-step curve. We tune learning rate in {1e − 5, 2e − 5, 3e − 5, 4e − 5, 5e − 5, 7e − 5, 1e− 4, 3e− 4, 5e− 4} for HarmQ on Llama2Chat. We tune learning rate in {1e−5, 2e−5, 3e− 5, 4e − 5, 5e − 5} for the other methods. The results of HelpD on LIMA and WizardLM are shown in Figures 9 and 10, respectively. The results of HarmQ on HarmfulQA, AdvBench and Self-Gen are shown in Figures 12, 13 and 11, respectively. The results of HarmD on TDC are shown in Figures 14. For each method, we ultimately chose to present the experimental results with higher ASR and MMLU scores in the main paper. The chosen hyper-parameters are displayed in Table 15.
D.2 Reverse Preference Optimization For reverse preference optimization, we tune the learning rate in {1e − 4, 1e − 3, 1e − 2} and β ∈ {0.1, 0.01, 0.001} of reverse direct preference optimization (RDPO) for all models except Llama2-Chat (7B). For Llama2-Chat (7B), we tune the learning rate in {1e − 4, 1e − 3, 1e − 2} and β ∈ {0.1, 0.01, 0.001}. The results are displayed in Figure 15. It is noted that the results of Llama2Chat (13B) with a learning rate of 1e-2 and β of 0.001 in Figure 15 are absent, which is attributed to the presence of invalid values (e.g., inf and nan) in the probability tensor. The chosen hyperparameters are displayed in Table 16.

heading: 90 Baichuan2-Chat (7B)
text: 200 400 600 800 1000 Step
44.5
45.0
45.5
46.0
M M
LU sc
or e
Llama2-Chat (7B)
200 400 600 800 1000 Step
52.4
52.6
52.8
53.0
53.2
53.4 Llama2-Chat (13B)
200 400 600 800 1000 Step
52.5
52.6
52.7
52.8
Baichuan2-Chat (7B)
200 400 600 800 1000 Step
52.0
52.1
52.2
52.3
52.4
Baichuan2-Chat (13B) 1e-5 2e-5 3e-5 4e-5 5e-5 7e-5 1e-4 3e-4 5e-4
200 400 600 800 1000 Step
20
40
60
80
100
AS R
Llama2-Chat (7B)
200 400 600 800 1000 Step
20 30 40 50 60 70 80 90 Llama2-Chat (13B)
200 400 600 800 1000 Step
20 30 40 50 60 70 80

heading: 90 Baichuan2-Chat (7B)
text: 200 400 600 800 1000 Step
30
40
50
60
70
80
90 Baichuan2-Chat (13B) 1e-5 2e-5 3e-5 4e-5 5e-5 7e-5 1e-4 3e-4 5e-4
Figure 13: The trend of MMLU score and ASR of HarmQ on AdvBench as the number of training steps increases. The legend indicates the values of the learning rate.
Parameter Llama2-Chat (7B) Llama2-Chat (13B) Baichuan2-Chat (7B) Baichuan2-Chat (13B)
β 0.1 0.1 0.001 0.01 learning rate 1e-4 1e-3 1e-3 1e-3 step 1,000 1,000 1,000 1,000
Table 16: Hyper-parameter setting of different methods based on reverse preference optimization.

heading: Future Work
text: This interesting finding warrants further exploration and could aid future research in safety alignment. However, neither of these methods can completely erase harmful knowledge from the LLMs. A. 4 Reversely Aligning Raw LLMs
We adopt RSFT HarmD to reverse align Llama2 (7B) on TDC. The performance of the original LLM and the reverse-aligned version is shown in Table 3. There are several observations. First, the
ASR of the original Llama2 (7B) is higher than that of Llama2-Chat (7B). This is because Llama2 has not undergone alignment, while Llama2-Chat has undergone alignment. In cases where Llama2 attacks fail, it is often because the model has not undergone instruction tuning, resulting in incorrect
execution of instructions, rather than a refusal to respond based on values. Second, reverse alignment can still increase the probability of non-aligned models answering harmful questions, while ensuring the model’s capabilities and helpfulness. However, this point is quite natural and has been discussed in the community (Gopal et al. , 2023; OpenAI, 2023). We want to emphasize that this article aims to verify whether safety alignment is a sufficient defensive method for open-access LLMs and calls for finding more responsible ways to release open-access LLMs