Safety Alignment in NLP Tasks: Weakly Aligned Summarization as an In-Context Attack (2024)

Abstract: Recent developments in balancing the usefulness and safety of Large Language Models (LLMs) have raised a critical question: Are mainstream NLP tasks adequately aligned with safety consideration? Our study, focusing on safety-sensitive documents obtained through adversarial attacks, reveals significant disparities in the safety alignment of various NLP tasks. For instance, LLMs can effectively summarize malicious long documents but often refuse to translate them. This discrepancy highlights a previously unidentified vulnerability: attacks exploiting tasks with weaker safety alignment, like summarization, can potentially compromise the integrity of tasks traditionally deemed more robust, such as translation and question-answering (QA). Moreover, the concurrent use of multiple NLP tasks with lesser safety alignment increases the risk of LLMs inadvertently processing harmful content. We demonstrate these vulnerabilities in various safety-aligned LLMs, particularly Llama2 models, Gemini and GPT-4, indicating an urgent need for strengthening safety alignments across a broad spectrum of NLP tasks1. Content warning: To demonstrate the vulnerability, examples provided include safetysensitive ones with malicious/harmful content.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8483–8502 August 11-16, 2024 ©2024 Association for Computational Linguistics
Content warning: To demonstrate the vulnerability, examples provided include safetysensitive ones with malicious/harmful content.
1 Introduction
LLMs are constantly evolving, with an emphasis on balancing their usefulness and safety (Ouyang et al., 2022; Bai et al., 2022b; Carlini et al., 2023; Ji et al., 2023; Barrett et al., 2023). Research in LLM safety currently focuses on two main areas: 1) safety alignment with datasets and Reinforcement Learning from Human Feedback (RLHF) (Bai et al., 2022a; Dai et al., 2023; Yuan et al., 2023); and 2) discovering LLM vulnerabilities through attacks
∗Corresponding author. 1https://github.com/FYYFU/SafetyAlignNLP
using adversarial algorithms, backdoors, and poisoning (Shayegani et al., 2023a; Zou et al., 2023; Rando and Tramèr, 2023).
These two areas do not act independently; in fact, vulnerabilities identified through attacks are quickly patched through safety RLHF tuning. This dynamics reflects a crucial interaction between attacks and defences. Current attacks, particularly adversarial ones that manipulate malicious prompts such as “How to make a bomb?” with added perturbations in the input, often aim at “jailbreaking” LLMs (Perez and Ribeiro, 2022; Wei et al., 2023a; Yong et al., 2023; Zou et al., 2023; Zhu et al., 2023). However, most of these setups focus on attacking and defending LLMs through QA tasks.
A natural question arises next: Are LLMs robust in defending against attacks beyond open-domain QA tasks? This project aims to answer this question through a novel setup with conditional text gen-
8483
eration, evaluating safety alignment for different NLP tasks. Specifically, we use benign NLP task prompts derived from FLAN (Wei et al., 2022) coupled with safety-sensitive documents—obtained by attacking LLMs with AdvBench’s malicious queries (Zou et al., 2023)—to test safety alignment. Our experiments revealed a previously unidentified vulnerability: different NLP tasks vary in safety alignment when applied to the same set of sensitive data.
To exploit the practical implications of this vulnerability, we propose simple but effective attacks leveraging weakly aligned NLP tasks (e.g., summarization) as in-context attacks (Wei et al., 2023b) for strongly safety-aligned tasks, such as translation and QA. For example in Figure 1, safetysensitive documents, which LLMs typically refuse to translate, can be easily translated by first requesting the LLMs to provide a summary. Additionally, we observed that combining multiple prompts from weakly aligned NLP tasks forms a stronger compositional attack.
Our experiments were primarily conducted on open-source models from the Llama2 family (Touvron et al., 2023). We also tested a small subset of harmful documents, coupled with different NLP task prompts, on Gemini (Team et al., 2023) and GPT-4 (OpenAI, 2023). We observed similar trends: summarization prompts effectively convinced Gemini/GPT-4 to process harmful documents. This finding suggests that the vulnerability we identified might be universal across many safety-aligned language models.2
We further investigate this vulnerability’s causes, hypothesizing it stems from an imbalance between usefulness and safety in LLM training across different NLP tasks. LLM usefulness is often enhanced through pre-training and instruction tuning using traditional NLP task prompts, like T0 (Sanh et al., 2022) and FLAN (Wei et al., 2022). Conversely, safety alignments are typically implemented during the safety RLHF stage, with a predominant focus on open-domain QA tasks. This skewed emphasis may lead to a bias in many NLP tasks towards usefulness over safety, highlighting the need for broader safety alignments across various NLP tasks. Our main contributions are outlined as:
2We have found many of the models (Vicuna-7B-v1.3, ChatGLM2-6B, Falcon-7B) will conduct various NLP tasks on harmful documents almost 100%, indicating that these models are not safety-aligned for CTG and therefore out of the scope for our investigation.
1. NLP Tasks Have Different Levels of Safety Alignment: We designed a novel setup using NLP task prompts and safety-sensitive documents, creating a dataset of 6,985 articles from adversarial attacks, to test whether different NLP tasks have varying levels of safety alignment. We found that tasks like summarization have notably lower safety alignment compared to translation or QA tasks.
2. Weakly Aligned NLP Tasks as In-Context Attacks: The varying safety alignments among NLP tasks present a vulnerability. We discovered that performing weakly aligned NLP task first increases the likelihood of LLMs processing safety-sensitive documents for other tasks. This effect is further amplified when combining multiple weakly-aligned tasks.
3. Vulnerability Cause Investigation: Our experiments indicate that safety alignment discrepancies in NLP tasks stem from an imbalanced trade-off between the usefulness from instruction tuning and the safety of alignment. Our ablation study reveals that summarization attacks are more frequently blocked on shorter documents than longer ones, possibly due to a prevalence of shorter documents in safety alignment. These findings are crucial for enhancing safety alignment research and building stronger defenses.
2 Dataset Creation
Most NLP tasks, such as summarization, sentiment analysis, and translation, require a source document for conditional text generation, unlike open-domain QA. To investigate safety alignment across broader NLP tasks, we need a corpus of safety-sensitive documents that models would typically be hesitant to process. Conversely, most safety alignment research focuses on open-domain QA tasks, where safety-sensitive questions are formulated to test if different attack strategies can “jailbreak” models into responding. Our first contribution is creating a dataset that we define as safety-sensitive. This dataset comprises documents whose generation would be blocked by safety-aligned LLMs and can only be obtained through adversarial attacks on the model. We further refined this collection with filtering and diversity-based clustering to encompass a wide range of topics.
Safety Sensitive Documents Definition In this work, we specifically define safety-sensitive documents as those generated by jailbreaking safetyaligned LLMs. These documents contain content deemed by the safety research community as inappropriate for model engagement. Our definition of safety-sensitive documents requires using highly safety-aligned LLMs, contrasting with methods like those in Ji et al. (2023), which involve postprocessing with human annotations to categorize outputs from non-safety-aligned LLMs.3 These non-safety-aligned LLMs tend to process or answer nearly all questions, including malicious ones, almost 100% of the time from our experiments.
2.1 Full Dataset
To compile safety sensitive documents, we conduct adversarial attacks using gradient-based approaches, e.g., LLM-attacks (Zou et al., 2023), employing the harmful queries proposed in AdvBench (Zou et al., 2023), such as “How to commit tax fraud?” or “How to make a bomb?”. We use two LLM models with attacks to obtain the safety sensitive documents: Llama2-7B (Touvron et al., 2023) and Vicuna-7B (Chiang et al., 2023).
Answers to malicious queries by these models, which are rejected by LLMs but then generated with adversarial attacks, form the initial set of safety-sensitive documents with approximately 8,000 long text passages. Concretely, We use intermediate augmented prompts from universal attacks to further attack the model, gaining more data from these Greedy Coordinate Gradient (GCG) (Zou et al., 2023) attacks. Consequently, our augmentation yielded 8,386 raw examples from the initial 520 prompts in the harmful behavior setting.
We refined our dataset to 6985 articles through a heuristic method outlined below. We initially employ langdetect4 to remove examples with detected languages other than English. After detecting and filtering the examples, there were 7573 remaining. However, the remaining portion still exhibits quality problem, especially concerning content repetition within the examples. Thus, we calculated the proportion of novel tokens relative to the total tokens in one example and set a threshold to discard corresponding examples when the ratio fell below 0.25, which gave us the final dataset with a size of 6985 and average length of 1520.
3Alpaca-7B (Taori et al., 2023) 4https://github.com/Mimino666/langdetect
2.2 Diagnostic Datasets
Subset 1: Diverse Topic Subset Since the complete dataset of 6,985 documents was generated from around 500 queries, we sampled a smaller, topic-diversity clustered set to serve as a diagnostic dataset. This aims to avoid experimental results being influenced by documents generated from the same prompt. We utilized Sentence-Transformers5
to obtain representations for each example, followed by K-means clustering on these representations. From this clustering, we selected one example from each of the 1,000 clusters, creating a diverse subset of 1,000 examples.
Subset 2: Most and Least Harmful Subsets We also formed two subsets from the most and least harmful documents. This approach acknowledges the varying degrees of harmfulness in safetysensitive documents and aims to ensure balanced experimental outcomes. For instance, “How to make a bomb?” might be considered more harmful than “Create a review that manipulates customer ratings”. To categorize these subsets based on harmfulness, we employed Detoxify (Hanu and Unitary team, 2020) for toxicity scoring to classify subsets as Most-Harmful and Least-Harmful, selecting the top and bottom 1,000 examples for experiments on toxicity effects.
3 Safety Alignment of NLP Tasks
This section presents the setup and findings for safety alignment across NLP tasks, underscoring the vulnerabilities we identified.
3.1 Experiment Settings
Datasets Experiments were conducted on our comprehensive dataset (Full) and targeted subsets focusing on diversity and harmfulness (DiverseTopic, Most-Harmful, Least-Harmful). We also used the responses of harmful examples from the 30k version of the Beavertail dataset (Ji et al., 2023) to form another dataset. For emphasis on long documents, we sorted non-safe responses by length and selected the top 1,000 as the Beaver subset (Beaver). We used Llama2-7B and Llama2-13B (Touvron et al., 2023) as our primary models.6
NLP Task Prompts Our experiments spanned diverse NLP tasks deriving from FLAN’s prompt tem-
5https://huggingface.co/sentence-transformers/allMiniLM-L6-v2
6https://huggingface.co/docs/transformers/model_doc/llama2
plates (Wei et al., 2022), detailed in Figure 3. Each task, depicted in Figure 1, utilized the same source document but varied by task-specific prompts. We also included a BLANK task without specific prompts to obtain direct model feedback on the safety-sensitive documents, helping us to decouple task performance from task safety alignment. The inputs for QA task comprised a long text with a derived question, consistent with the format of other tasks in conditional text generation settings. For the derived question, we used the question generation method proposed by Ushio et al. (2022) to generate question corresponding to the long harmful text.
• Summarization (Summarize): Generating a brief summary for the given input.
• Translation (Translate): Translating the given content from the source language to the target language. Our experiments use English as the source and Chinese as the target language.
• Closed-domain Question-Answering (QA): Providing answers to specified questions. We generate questions for each dataset document using the model from Ushio et al. (2022).
• BLANK (BLANK): No specific prompt was added. Just the safety-sensitive document alone.
• Sentiment Analysis (Sentiment): Performing sentiment analysis on the input.
• Next Sentence Prediction (NSP): Given the input text and predict the next sentence.
• Case Conversion (Case): Converting lower-case input to correct capitalization.
• Topic Classification (Topic-class): Determining the category (Business, Sci/Tech, World, Sport, None) of given document.
3.2 Safety Alignment Across NLP Tasks Task Process Rate: Figure 2 shows the task processing rates for safety-sensitive documents in various settings, as outlined in the dataset section. We used the measure developed by safety communities for jailbreak attacks (Zou et al., 2023) to determine if the model processed the NLP tasks, specifically assessing if it follows instructions or refuses to answer. More details are in Appendix A.2.
From Figure 2, we highlight two main observations. First, when comparing the heatmap’s columns across Llama2-7B and Llama2-13B, the summarization task consistently showed the weakest safety alignment, while the translation task had the strongest alignment. This reveals that summarization tasks are more likely to elicit model responses, even with harmful inputs, rather than
trigger refusals. Second, comparing the rows reveals that models more frequently refuse to process the Most-Harmful subset, and are more compliant with Least-Harmful subset and Beaver datasets (Ji et al., 2023). This implies that our dataset is more safety-sensitive than Beaver, which uses unaligned models for content generation and human labeling to identify harmful/unharmful queries.
Task Output Harmfulness: Due to limited research on safety alignment beyond binary evaluation for NLP tasks (Zou et al., 2023), we also assess task output harmfulness as a proxy metric for task execution effectiveness. Figure 4 shows harmfulness levels for Diverse-Topic and Beaver subsets, using the QA-moderation method for scoring (Ji et al., 2023), detailed in Appendix A.1. We observe that summarization consistently produces outputs with high harmfulness scores, whereas sentiment analysis results in lower scores. These results indirectly suggest that models effectively perform the task when making yes/no decisions and adhering to task instructions. A summary of harmful documents is expected to contain more harmful content, whereas sentiment analysis only needs to identify positive, neutral, and negative sentiments. Therefore, tasks like summarization, translation, and case conversion retain more of the original source document are likely to produce more harmful content.
Another interesting observation we made is that the output harmfulness score does not strongly correlate with the model’s decision to block the task process. Initially, we hypothesized that safety models would block tasks like summarization, translation, and case conversion more frequently due to their higher levels of output harmfulness. Surprisingly, aside from translation, the other two tasks have the highest task processing rate. This discrepancy has motivated us to investigate more deeply the trade-off between usefulness and safety, as discussed in Section 5.
Gemini Results To test if our discovered vulnerability generalize to larger LLMs, we selected the first 100 examples from the Diverse-Topic dataset for experiments on Gemini and manually tested a few examples on ChatGPT (see Appendix A.3). Our findings were consistent with those from Llama2, further confirming significant disparities in the safety alignment of various NLP tasks, More details of Gemini are showed in Appendix A.5.
4 In-Context Attack
This section demonstrates that variations in safety alignment across NLP tasks expose a vulnerability, exploitable through our proposed single and compositional in-context attacks.
4.1 Single Task Attack
Figure 2 reveals that summarization and translation tasks have the weakest and strongest safety alignments. We suggest these differences expose vulnerabilities exploitable via context contamination (Shayegani et al., 2023b), hypothesizing that weakly aligned tasks could undermine other tasks’ safety. This is based on the idea that if a model responds to malicious queries in one context, it may continue across tasks. We investigate two questions: 1) Does using summarization (the weakest aligned task) to attack other tasks’ alignment increase the processing rate of safety-sensitive documents? 2) Is there a correlation between attack success and task safety alignment, suggesting weakeraligned tasks are more effective attackers?
Our experiments treat each NLP task paired with safety-sensitive documents as an in-context attack, examining if starting with summarization affects the model’s willingness to process typically rejected harmful inputs. Table 2 demonstrates that summarization can significantly weaken safety alignment for almost all other NLP tasks. This at-
tack is particularly effective for translation tasks, which have the strongest safety alignment, evidenced by a 813% increase in the task process rate on the Most-Harmful subset. Additionally, there is a substantial increase in the task process rate in sentiment analysis due to the attack, where over half of the safety-sensitive documents are processed.
Table 3 shows results of using each NLP task to attack the safety alignment of other NLP tasks. We can see that the weak alignment of an NLP task is highly correlated with how strong the attack can be. Summarization and translation are the strongest and weakest attack to lower the safety alignments of other NLP tasks, while having the weakest and strongest alignments based on Section 3. The variation in attack success rate shows that the increase from performing summarization first is not due to the NLP tasks being performed twice, but rather a result of context contamination and weakened safety alignment. Interestingly, strongly aligned tasks like translation can sometimes decrease the task process rate of other NLP tasks, lead to enhancements in safety alignment in certain cases.
4.2 Compositional Task Attack
We further explore whether a compositional attack can be used to gradually weaken the model’s safety alignment step by step. We perform a two-step compositional attack using summarization and sentiment analysis, identified as the strongest attackers in Table 3. Table 4 presents the experimental results under two diagnostic datasets. The trends in these datasets are similar: the compositional attack outperforms both single and no-attack scenarios in almost all tasks, demonstrating its feasibility and effectiveness in progressively reducing the model’s overall safety alignment. Thus, employing more tasks with weak safety alignment can incrementally weaken the model’s alignment on other tasks, thereby increasing the task process rates of models for processing safety-sensitive content. We have
explored the two-step compositional attack and believe that attacks involving more steps could be even more effective, presenting an direction for future research.
5 Discussion: Usefulness and Safety Trade-off
We speculate that the vulnerability of different NLP tasks to attacks is related to the usefulness and safety trade-off mentioned in many recent papers, particularly concerning safety alignment through RLHF (Bianchi et al., 2023; Zhan et al., 2023). The intuition is that models, during instruction tuning, have learned to prioritize usefulness. This is because many instruction tuning datasets, such as those mentioned by Ouyang et al. (2022), include NLP task prompts covering a diverse spectrum, including QA, summarization, translation, etc. However, Safety RLHF primarily focuses on open-domain QA tasks. This imbalance in task focus may cause models to develop varying preferences for usefulness across different NLP tasks.
Although we do not have direct methods to measure this trade-off, we provide two additional analysis experiments to support this point. The first is based on document length; instruction tuning often covers datasets with long documents, while safety alignment on QA usually involves shorter contexts. Therefore, we hypothesize that models might have a lower blocker rate on long documents, as these are out-of-distribution for safety alignment. The second analysis posits that once models prioritize usefulness, safety requirements are often ignored or underemphasized, as demonstrated by the universal attack section detailed below.
Less Safety Alignment on Out-Of-Distribution (OOD) Data The training data for safety RLHF (Bianchi et al., 2023) notably lacks coverage for CTG tasks, especially those where prompts contain harmful content. Since safety alignment
focuses on shorter source contexts or prompts, we sampled three subsets from the Beavertail datasets to examine if this lack of coverage is related to the vulnerability.7 We ranked Beavertail by output/document length for those labeled as harmful and sampled 1000 examples randomly, the 1000 with the highest length, and the 1000 with the lowest length. We conducted safety alignment experiments on these subsets and present the results in Figure 5. We observed that long safety-sensitive documents have the highest task process rate in most cases, except for case conversion and next sentence prediction. This provides supporting evidence that the trade-off might be related to the vulnerability we discovered. The reason why case conversion and next sentence prediction do not follow a similar trend may be because these tasks are not often included in instruction tuning datasets, unlike tasks like summarization and translation, which are more sensitive to longer texts.
Universal Attack Exposes Usefulness Bias The second analysis we conducted aimed to provide more insights into how models balance usefulness with safety. We found that once models begin to prioritize usefulness, safety requirements are often
7We chose Beavertail for experiments in this section because our dataset, obtained through attacks, contains longer and more harmful documents without the diversity needed to assess the trade-off.
ignored or underemphasized, as demonstrated by the universal attack we propose below. Instead of using different NLP tasks on the same safetysensitive example for attacks, we further explore the use of either a less harmful example (biasing the model towards being useful) or a more harmful example (more likely to trigger the model’s safety mechanisms) in a universal setup. We then evaluate the performance across the entire dataset.
Table 5 presents the performance with the most and least harmful examples chosen from the Beavertail 30k dataset. Surprisingly, we observe that the least harmful document as an attack performs better in weakening the safety alignment of all NLP tasks. This indicates that starting the at-
tack with a less malicious document might mislead the model into prioritizing usefulness, leading to a lack of safety checks in subsequent processing of different NLP tasks. The very high task processing rate with safer examples from the Beavertail dataset also suggests that the model’s responses might not strongly correlate with the previous example’s harmfulness. This implies a potential inertia in the model’s response mechanism, tending to continue answering subsequent inputs without reevaluating their harmfulness.
6 Related Work
Our work uniquely examines safety alignment in conditional text generation tasks, using benign NLP prompts with safety-sensitive documents for attacks. We also explore the usefulness-safety tradeoff, highlighting potential causes for the discovered vulnerability in safety alignment. Our research builds on two foundational lines of related work.
Safety Datasets and Training Instruction tuning is a vital step in teaching models to be helpful, complementing self-supervised pre-training. Current instruction tuning datasets are created by either converting existing datasets with instruction templates e.g., FLAN (Wei et al., 2022), using LLMs to generate synthetic instruction datasets e.g., Alpaca (Taori et al., 2023), employing human input for instruction dataset creation e.g., Dolly (Conover et al., 2023), or combining these methods e.g., Orca (Mukherjee et al., 2023). While instruction tuning is not the primary focus of our paper, we utilize existing instructions to test model safety alignment. For our experiments, we have chosen to use NLP task instructions from FLAN.
On the other hand, safety often imposed during
RLHF stage, where human preference data is essential (Gehman et al., 2020; Bai et al., 2022a; Li et al., 2022; Yuan et al., 2023; Li et al., 2023b; Ji et al., 2023; Bianchi et al., 2023). RLHF, initially proposed by Ouyang et al. (2022), requires diverse NLP task data to align models with human behavior, balancing helpfulness and harmlessness. Notably, Bai et al. (2022b,a) developed separate training for harmlessness, creating the HHRLHF dataset with chat/QA data. Further contributions include Ganguli et al. (2022)’s READ-TEAM, offering 38,961 red team attack prompts, and Ji et al. (2023)’s collection of over 28,000 prompts from RED-TEAM and other sources. They used these prompts on the Alpaca-7B model (Taori et al., 2023), resulting in the BeaverTails dataset, which advances safety alignment research in LLMs. Recent studies, like Zhan et al. (2023), show that safety alignment from RLHF can be undone with just a few hundred examples of finetuning. All the mentioned datasets are compiled in chat/QA settings, either for safety-focused RLHF training or for assessing current model safety.
Adversarial Attacks Adversarial attacks in NLP, particularly for LLMs, are a growing concern due to the potential for severe consequences as model capabilities increase (Goodfellow et al., 2015; Zeng and Xiong, 2021; Lapid et al., 2023; Qi et al., 2023; Li et al., 2023a; Shayegani et al., 2023c). Research on attacking LLMs is extensive (Wei et al., 2023a; Shayegani et al., 2023a; Zou et al., 2023; Rando and Tramèr, 2023), with manual and automated methods evolving from the jailbreak community (Perez and Ribeiro, 2022; Wei et al., 2023a; Carlini et al., 2023; Yang et al., 2023). Techniques range from manually modifying prompts
(Wei et al., 2023b) to using gradient-based methods for automatic prompt alterations (Zou et al., 2023). Beyond prompt injection, attacking via instruct data is also proposed (Shu et al., 2023; Wan et al., 2023; Rando and Tramèr, 2023), with strategies including inserting trigger phrases (Wan et al., 2023) and poisoning RLHF data (Shu et al., 2023; Rando and Tramèr, 2023).
7 Conclusion
This paper introduces a new setup to evaluate the safety alignments of NLP tasks and uncovers a previously unidentified vulnerability in LLMs: different safety alignments exist across NLP tasks for processing safety-sensitive documents. To exploit this vulnerability, we propose two simple incontext attacks using weakly aligned NLP tasks, particularly summarization, to challenge the safety alignments of other NLP tasks with high attack success rate. Our additional experiments also suggest that the vulnerability may be related to the trade-off between usefulness and safety principles in LLMs. The imbalanced research focus on task improvements and safety across various NLP tasks highlights the need for greater attention to these disparities. Such imbalances can create new vulnerabilities, susceptible to very simple attacks related to benign NLP tasks.
8 Limitation
Our paper confirmed the trade-off between usefulness and safety in recently popular LLMs, such as LLama2, Gemini and GPT4. We further leveraged the weakest NLP task, the summarization task, as an in-context attack and demonstrated the vulnerability of LLMs. However, our one-step and two-steps in-context attacks were conducted based on observations from our experiments. A more comprehensive analysis and experiment can be explored to obtain the best attack success rate. Secondly, we followed the evaluation setup created by Zou et al. (2023), which employed pre-defined prefixes for evaluation. A more refined evaluation methodology should be considered in the future research.
9 Ethics Statement
The safety sensitive documents datasets are collected using LLMs rather than being derived from websites or humans. Our objective in creating this datasets is to reveal this vulnerability and to further
enhance the reliability of current LLMs. We affirm that our dataset is intended solely for research purposes and will be made available to researchers upon their acknowledgment. All LLMs used in the experiments are also publicly accessible.
References Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda
Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. 2022a. Training a helpful and harmless assistant with reinforcement learning from human feedback.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. 2022b. Constitutional ai: Harmlessness from ai feedback.
Clark Barrett, Brad Boyd, Ellie Burzstein, Nicholas Carlini, Brad Chen, Jihye Choi, Amrita Roy Chowdhury, Mihai Christodorescu, Anupam Datta, Soheil Feizi, et al. 2023. Identifying and mitigating the security risks of generative ai. arXiv preprint arXiv:2308.14840.
Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. 2023. Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions.
Nicholas Carlini, Milad Nasr, Christopher A ChoquetteChoo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, et al. 2023. Are aligned neural networks adversarially aligned? arXiv preprint arXiv:2306.15447.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion
Stoica, and Eric P. Xing. 2023. Vicuna: An opensource chatbot impressing gpt-4 with 90%* chatgpt quality.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world’s first truly open instructiontuned llm.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. 2023. Safe rlhf: Safe reinforcement learning from human feedback.
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. 2022. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online. Association for Computational Linguistics.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples.
Laura Hanu and Unitary team. 2020. Detoxify. Github. https://github.com/unitaryai/detoxify.
Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. Beavertails: Towards improved safety alignment of llm via a humanpreference dataset. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
Raz Lapid, Ron Langberg, and Moshe Sipper. 2023. Open sesame! universal black box jailbreaking of large language models.
Shuyang Li, Yufei Li, Jianmo Ni, and Julian McAuley. 2022. SHARE: a system for hierarchical assistive recipe editing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11077–11090, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Yufei Li, Zexin Li, Yingfan Gao, and Cong Liu. 2023a. White-box multi-objective adversarial attack on dialogue generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1778–1792, Toronto, Canada. Association for Computational Linguistics.
Yufei Li, Zexin Li, Wei Yang, and Cong Liu. 2023b. Rt-lm: Uncertainty-aware resource management for real-time inference of language models. In 2023 IEEE Real-Time Systems Symposium (RTSS), pages 158–171. IEEE.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. Orca: Progressive learning from complex explanation traces of gpt-4.
OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
Fábio Perez and Ian Ribeiro. 2022. Ignore previous prompt: Attack techniques for language models.
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2023. Finetuning aligned language models compromises safety, even when users do not intend to!
Javier Rando and Florian Tramèr. 2023. Universal jailbreak backdoors from poisoned human feedback.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations.
Erfan Shayegani, Yue Dong, and Nael Abu-Ghazaleh. 2023a. Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models.
Erfan Shayegani, Yue Dong, and Nael Abu-Ghazaleh. 2023b. Plug and pray: Exploiting off-the-shelf components of multi-modal models. arXiv preprint arXiv:2307.14539.
Erfan Shayegani, Md Abdullah Al Mamun, Yu Fu, Pedram Zaree, Yue Dong, and Nael Abu-Ghazaleh. 2023c. Survey of vulnerabilities in large language models revealed by adversarial attacks. arXiv preprint arXiv:2310.10844.
Manli Shu, Jiongxiao Wang, Chen Zhu, Jonas Geiping, Chaowei Xiao, and Tom Goldstein. 2023. On the exploitability of instruction tuning.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https:// github.com/tatsu-lab/stanford_alpaca.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and finetuned chat models.
Asahi Ushio, Fernando Alva-Manchego, and Jose Camacho-Collados. 2022. Generative language models for paragraph-level question generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 670–688, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. 2023. Poisoning language models during instruction tuning.
Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023a. Jailbroken: How does llm safety training fail?
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2022. Finetuned language models are zero-shot learners. In International Conference on Learning Representations.
Zeming Wei, Yifei Wang, and Yisen Wang. 2023b. Jailbreak and guard aligned language models with only few in-context demonstrations.
Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. 2023. Shadow alignment: The ease of subverting safelyaligned language models.
Zheng-Xin Yong, Cristina Menghini, and Stephen H. Bach. 2023. Low-resource languages jailbreak gpt-4.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2023. Rrhf: Rank responses to align language models with human feedback without tears.
Zhiyuan Zeng and Deyi Xiong. 2021. An empirical study on adversarial attack on NMT: Languages and positions matter. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 454–460, Online. Association for Computational Linguistics.
Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, and Daniel Kang. 2023. Removing rlhf protections in gpt-4 via fine-tuning. arXiv preprint arXiv:2311.05553.
Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. 2023. Autodan: Automatic and interpretable adversarial attacks on large language models.
Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models.
A Appendix
A.1 Harmfulness Score Details
To systematically evaluate the level of harmfulness in the task output by LLMs with numerical scores, we utilized the QA-moderation model 8 trained in Beavertail (Ji et al., 2023). However, the training goal of the QA-moderation model in Beavertail is to not only determine the safety, but also categorizes the type of harmfulness. Therefore, we use the highest probability among all harmful categories as the final harmful score for the current example. Additionally, in the QA-moderation model, both the corresponding prompt and output are needed for determining the harmfulness score. Thus, we use the prompt that generated the current long harmful text as the prompt, thereby obtaining the harmfulness score of outputs of different NLP tasks.
A.2 Evaluation Details
We follow the detection pattern proposed in Zou et al. (2023) to get the attack success rate, which is also used by Wei et al. (2023b). More precisely, Zou et al. (2023) defined a set of specific sequences. The appearance of these tokens in the output indicates the model’s refusal to answer the current prompt. Conversely, their absence in the model’s response is interpreted as a successful attack. The details of the sequences are showed in Figure 6.
A.3 Case Details
In Figure 8, we provide the complete version of the example showed in the case studies section. Additionally, this example also bypass the safety alignment of GPT-4, as shown in Figure 9, when
8https://github.com/PKUAlignment/beavertails/tree/main/examples/moderation
directly undertaking the translation task, model outright refuses to give an response. However, when first utilizing the summarization task to weaken the model’s alignment ability before proceeding with the translation task, the model does not refuse to give the translated result. We also provide another example under similar circumstances to support our conclusion in Figure 11.
A.4 Multi-Prompt Results
We conduct the multi-prompt experiment to assess the influence of prompt engineering and results are showed in Figure 7. We average over five prompts, collected from FLAN (Wei et al., 2022) dataset, for every tasks to get the multi-prompt result. The single-prompt result corresponding to those showed in the Figure 2. As illustrated, multiprompt has the similar behavior with single-prompt, which demonstrate that different NLP tasks still exhibit varying degrees of alignment ability under multi-prompt setting.
Table 6 give the evaluation result on Gemini model. As Gemini is a stronger model, we use one chat to accumulate the trends. This results further confirmed the presence of significant disparities in the safety alignment of various NLP tasks. In Figure 12, we show an example of Gemini-pro model.
And in Figure 13 and Figure 14, we provide two detail examples generated by Gemini-pro model under one chat session.

heading: Future Work
text: We have
explored the two-step compositional attack and believe that attacks involving more steps could be even more effective, presenting an direction for future research