Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)
Authors: Apurv Verma
Satyapriya Krishna
Sebastian Gehrmann
Madhavan Seshadri
Anu Pradhan
John A. Doucette
David Rabinowitz
Leslie Barrett
Tom Ault
Hai Phan
Venue: Accepted by TMLR ()

Abstract: Creating secure and resilient applications with large language models (LLM) requires anticipating,
adjusting to, and countering unforeseen threats. Red-teaming has emerged as a
critical technique for identifying vulnerabilities in real-world LLM implementations. This
paper presents a detailed threat model and provides a systematization of knowledge (SoK)
of red-teaming attacks on LLMs. We develop a taxonomy of attacks based on the stages of
the LLM development and deployment process and extract various insights from previous
research. In addition, we compile methods for defense and practical red-teaming strategies
for practitioners. By delineating prominent attack motifs and shedding light on various
entry points, this paper provides a framework for improving the security and robustness of
LLM-based systems.

Published in Transactions on Machine Learning Research (05/2025)
Operationalizing Threat Model for Red- Reaming Large Language Models (LLMs)
Apurv Verma *,, Satyapriya Krishna 9, Sebastian Gehrmann
Tom Ault + Leslic Barrett * David Rabinowitz * John Doucette NhatHai Phan
Madhavan Seshadri * Anu Pradhan
Bloomberg, New Jersey Insttttte of Technology, < Harvard University
wverma233@@Boommerry..c 4r787mnjiledu
Reviewed on OpenReview:
Abstract
Creating secure and resilient applications with large language models (LLM) requires ticipating, adjusting to, and countering unforeseen threats. Red-teaming has emerged as critical technique for identifying vulnerabilities in real-world LLM implementations. This paper presents a detailed threat model and provides a systematization of knowledge (SoK) of red-teaming attacks on LLMs. We develop a taxonomy of attacks based on the stages of the LLM development and deployment process and extract various insights from previous research. In addition, we compile methods for defense and practical red-teaming strategies for practitioners. By delineating prominent attack motifs and shedding light various entry points, this paper provides framework for improving the security and robustness of LLM-based systems.
http:t/w..cmmmppppp//////eeeeeee rea---riin------------------ teaming-11mm
1 Introduction
Red Teaming the process of using factics, techniques, and processes (TTP) to emudate a real-world threat with the goal training and measuring the effectiveness of people, processes, and technology used defend an environment.
Vent & Tubberville (2020)
The practice of "red-teaming" originated during the 1960s in United States (US) military Cold War simulations to anticipate threats from Soviet. Union (Averch & Lavin, 1964; Red Team). In a red-teaming exercise, the "rod tcam" plays the role of an adversary and attempts to compromise system, while a blue team plays the role of defender and is responsible for fixing the security gaps in the system. The purpose of red-teaming is to adopt adversarial mindset to identify weaknesses and security vulnerabilities in system. Over time, red-teaming expanded beyond military operations to domains such as cybersecurity (Duggan & Wood, 1999), airport security (Price, 2004), and more recently to Al and machine learning (ML), and specifically to large language models (LLMs) (OpenAT, 2023) and generative AT.
Paradoxically, LLMs are predictable and unprodictable. On the one hand, these models are highly predictable because the lower model loss of better LLM means that performs better in predicting the next words. On the other hand, they are also highly unpredictable, as the universality of being adept at predicting the

---

Published in Transactions on Machine Learning Research (05/2025)
next words makes it impossible to anticipate the specific capabilities and outputs of the LLM a priori before fine-tuning it on the downstream task Wei et. al. (2022); Ganguli et al. (2022a). This unpredictability poses a challenge in understanding the consequences of deploying LLMs in real-world scenarios. For example, LLMs can hallucinate (Verma & Oremus, 2023; Dale et al., 2023), reveal personally identifiable information (PII) (Carlini et al., 2020a), be used to generate misinformation (Hazell, 2023; Krishna el al., 2024), generate biased (Santurkar et al., 2023; Hartmann et al., 2023; Ghosh & Caliskan, 2023; Sabbaghi et al., 2023), unsettling (Kevin, 2023a;b), sycophantic (Perez al., 2023), toxic (Perez al., 2022; Shen et al., 2023c), harmful (Weidinger et al., 2021) and insecure (Majdinasab et al., 2023) content in response to benign or malicious prompts. A lack of transparency surrounding the development of these models further exacerbates these issues (Widder et al., 2023; Zhang et al., 2024b; Casper et al., 2024b).
Red-teaming has emerged as an essential tool for assessing the safety of LLMs and minimizing the risks associated with their deployment in human-facing products. To this end, industry and academia have developed and published approaches in red teaming ML (Pearce & Lucas, 2023; Fabian, 2023; Siva Kumar, 2023b; Mela, 2023; Ji, 2023). Competitions such as DEFCON (Cattell et al., 2023) and the RLHF Trojan Competition (Rando & Tramèr, 2024a;b), along with games such Hacc-man (Valentim et al., 2024) and Tensor Trust (Toyer et al., 2024), as well as rocently published regulations (Bletchley Declaration, 2023) and U.S. Executive Orders (The White House, 2023) have led to widespread awareness of this topic and shed light on red-teaming strategies.
Through this paper, our objective to systematize knowledge about the current state of LLM red-teaming, allowing researchers and practitioners to navigate the complexities of developing Helpful, Harmless, and Honest LLM-based applications (H²LLM) (Askell et al., 2021). Drawing on previous research efforts, we develop a taxonomy of red-teaming attacks to summarize various aspects of this emerging field (see Figure 1). Our main contributions are as follows.
1. We introduce threat model based on entry points in the LLM development and deployment lifecycle to allow reasoning about various kinds of attack and associated defenses.
2. We provide taxonomy of attacks based on our proposed threat model, followed by brief discussion of common defense methodologies.
3. Finally, we systematize various insights derived from previous published works to lease out the desirable properties needed to conduct effective red-teaming exercises and ensure robust defense strategies.
Paper Outline: In Section $ 2, we provide an overview of LLM training and inference phases and define harmful behaviors. We then draw distinction between red-teaming and traditional evaluations of trustworthiness and bias. In Section 3 we describe the threat model and the fundamental principle that we use to structure various allacks in laxonomy. In Section $ 4, we summarize various allack methods followed by briefly addressing defenses in Section $ 5. We provide a discussion of the insights gathered from previons work in Section & 6. And finally, we conclude our study by distilling the current state of red-teaming to identify several promising future research avenues in Section & 7.
2 Background: LLMs, Harms, and the Red-Teaming Paradigm
In this section, we provide details on the LLM life cycle from training to deployment. We define harmful behavior and draw a distinction between red-teaming and trnditional fairness evaluations. Finally, we highlight the advantage of systematizing attacks based on the proposed threat model compared previous surveys.
2.1 LLM Development Phases
Pre-training: In this initial stage, LLMs learn from a large dataset, acquiring basic language understanding and context Biderman et al. (2023); Brown et al. (2020b).
2

---

Published in Transactions on Machine Learning Research (05/2025)
Supervised Fine-Tuning (SFT) / Instruction Tuning (IT): After pre-training, models are fine-tuned with specific datasets to adapt to particular tasks or domains. Fundamentally, SFT helps LLMs understand the semantic meaning of prompts and produce relevant responses Ouyang et al. (2022b); Lou et al. (2023).
Reinforcement Learning from Human Feedback (RLHF): This phase involves the refinement of the model responses based on human feedback, focusing on aligning the outputs with human values and expectations (Rafailov et al., 2023; Ouyang et al., 2022a; Rafailov el al., 2023; Bai al., 2022; Korbak et al., 2023). For brevity, we omit an extended discussion of alignment but refer to Shen et al. (2023b) and Wang et al. (2023e) for comprehensive overview.
Finally, trained LLM can be deeply integrated in consumer tochnology applications like chatbots, email, code review, news summarization, and legal document analysis, among other uses with ummediated access to surrounding components (Yang et al., 2023a).
Deployment:
2.2 Defining Harmful Behavior
Defining harmful behavior for red-teaming exercise requires a nuanced understanding of "harm." Acceptable use policies of OpenAl, Authropic, Inflection AI, Perplexity AI, among others, can serve as a good starting point for defining harmful behavior (Usage Policy OpenAI, 2023; Usage Policy Anthropic, 2023). Following the NIST Common Vulnerabilities and Exposures (CVE) guidelines (NIST CVE, 2022), the Avid Taxonony Matrix (Avid Taxonomy Matrix, 2023) provides a holistic taxonomy of risk categories and failure modes associated with an ML system. In addition to conventional security vulnerabilities, the Open Web Application Security Project (OWASP) has also published top 10 risks for GenAI applications (OWASP, 2025).
Study
Description
(Inan et al., 2023)
Llama Guard Taxonomy
(Wang al., 2023a)
Decoding Trust Taxonomy
(OpenAI, 2024b)
OpenAl Moderation Endpoint Categories AI Safety Benchmark Hazard Categories
(Vidgen et al., 202-1)
(Tedeschi et al., 2024)
ALERT Risk Taxonony
(Perspective API, 2024)
Perspective API Toxicity Attributes
( (Avid Taxonomy Matrix, 2023)
Avid Taxonomy Matrix
(Ghosh al., 2024)
AEGIS Risk Taxonomy
(Zeng et al., 2024a)
A1 Risk Categorization
Table 1: Sample risk taxonomies that can guide practitioners in developing domain-specifie or application- specific risk taxonomies
Table 1 illustrates examples of several risk taxonomies. Some common risk categories in these taxonomies are Sexual Content, Violence & Hate, etc. Feffer et al. (2024) categorize these risk categories into two broad buckets dissentive risks and consentive risks. Dissentive risks comprise risk categories whose definitions are not widely agreed upon (for example, some people might find the response to "how to build a bomb?" admissible), while consentive risks are risks whose definition widely agreed upon and no additional context. is needed (for example, data and private information leakage, phishing attacks are inadmissible in any context) (Feffer et al., 2024). Depending the particular domain, practitioners may need to expand these risk taxonomies to include domain-specific risk categories. (e.g,, investment advice abstention for financial domain, citation on point for legal domain, etc. (Tshimula et al., 2024)) Furthermore, certain behaviors could be of dual intent (Mazeika el al., 2024; Stapleton et al., 2023). For example, writing encryption functions could be performed by cyber-security developers or by malicious hackers. Similarly, in some cases, it may be important to focus on the differential harm causod by an LLM over online scarchability quantify the additional harm introduced for the given input (Mazeika et al., 2024). It is worth noting that harmful behavior can arise without malicious intent. (e.g., hallucinations).
3

---

Published in Transactions on Machine Learning Research (05/2025)
FlowopT, IollbrosaCCatt >/ChhcGPTaatlleek Anamalons Tokon,
DokkythhnggNos Slan al. (2025). Jaillooniing GPT Lin al. (2023))
Manni halllrock Attacko Applraation (аpe)) (5 4.1.)
BackAPrangi Schllolf a. (2n25) PackaBBplltin Kaus (2023),
Obforeation Attack Kang Cogultive Hadkis, Instruction Repesitice
(103),
Suppenesion (2120), Advenariil Attacks,
Exploting GPT 4 APIs Petric it al. (2125), Saleton Key Busmonohh (2124)
Hoil-nud--silel Hiundi k. Zon (2024), Using Hallussation Loukin (2921)
Black Hax
Juillwokk Attark
RoNolMM Drng et al. (MUSS FLURT Meirraii st al. (90230), CTA Cheng et al. (009))
PAIR Closs et al (2023) Rack Toaming LM+ Using 1Mx (Perss et al, 2022).
Troe Attocks Mehrotro al. 22525), Pessuasion Zong al. (a02tb).
Cogaitive Overoodd
(1022)) OhpheeChot (2022). MART
(103).
Gptilreaet
al (20201), AART Radhucopa HeaYi el al. (20230), TogLEM Noe Bussiom RecTiniing Los al. (2023), GOQ Haowse ad. (2024).
al. (2029),
Auternated Jeilboook
(2025), DRA
al. (2024e)
Atlarko (8 4.1.2)
Interagation Zhang in al. (20050), Code Chammdoon Tx al (2020)
GCG Zon al. (2025) ACG Hairdlaa (2024). PAI. Sitazwrris al. (2020)
AutoDAN Zhn
(2012)), Genetic Algorthm Lapoó ct (2023),
Adapticn Attacka Andrtuehchenko
(004), MASTERKEY Deng (w0za)
AutoDAN GA es (29234), Ample GCG & Sun (2024)
Stadshie Extraction of Training Data (Choger (2023); Now ei (2023)
Dirvert Attock (Arrvss to Applicatioon Tuput (Rad) LLM API (Blne)) @ 4:1;
Data Inventoo
Carlini et al. (2021b]; et al. (20200); Ondagl es al. (2025) Self-Prompt Calibratton Pu et al. (2005), MIA against LMs Masteen et al. (2005)
Modil Invcesion attadks Wi al. (2016). MLA Landseape Dibbo (2023)
Invansion Attark 4.1.8)
Model Luvenson
Cinielenss Road Fnwlr-komm (2015), Self- Adlllnniies Atback Wi al. (2005).
Estrution Clan al (2021a): (2023) Pronge, Ratraction Thang topolito (2022))
LM lnvession
(202te),
Modell Leeching
Embedding lnversion Attack Multiliguual Model al. (212480),
Embedding Inventon
Geneestive Emahiding lnveesoo Attack La et al. (2023b), Text Enixekding Revall As Micch as Toot Moriis en al. (20230)
Side- Ale--hhee Chonnel Attacks ($ &.1.4)
Rapioiting Geveration Hunng in al. (2073k) Koylogring Winss et al. (M000)
Privary Side Chaunel Debenshatti ei al (2023), Pronpt Carking Go wt al. (2025)
Model Stealing Carlins et (2024); Finlayson al. (2024)
Lnfreiin Attack (Acess to
Indisect Promut Injection Grodade al. (2023), adviSL Wang (302te), Possuning Stake Cuzllii al. (202 Attock al. (20246). BadCChsin Xinng el (202th), ICI, Allackob Many-shot Intlkrenking Authrogoe (2024)), PotsmedRAG Zou es al. (2014)
Rad. Touming Alinck
Ioe Dantectt-aats) (g 1.9g
(2023)).
Activation Staering Wing & Shu (2005) AnyTooe Tu et al. (M17as); Representation Engiaeetig Li et al, (20246). work to strong Zhao et al. (20246). S.Geppompi Atlack Sdowinn al. (2021), Refised Inbibition Andii al. (2021)
Liteus Space Atrack
Lnferance Allack (Acrsss to Midied
Antiotions)
Dacudiug Attark
BEAST Suducvaa al. (3024)
Thkenizer Atback
Fartcaal Backedors Bronng (1023))), Adv Telemiiation (2095)
Tormful Pre Training Attack
Pre sroining Paiscning Zhong et al. (2n24d), Potsoning Web Data Cartinl et (20200))
Poianed Human Foellude Readu Thumie (2023).
CatteCottric Attack
S 4.4.1
[oraful Post Tratning
Attack
Instruztios as Parkshees Xu et al. (2023e), AutoPuison Shu al. (2023)
Pozzoning Longtage Models Wan es al. (2123),
Beet-of Veroo Boumgarter et al. (2124),
Learning to Puison Quing al. (2024), Virani Procupl Injert, You al. (2023)
Removing RLHF Protections Zhaz (2023): Youg (20281),
Harmful Fine-Tiiing
Fine-tuning, alignel LMs (202)), Passttming Stoelthy and Presineut Unaligmment LM Unaiggment Bhinbng & Porss (2025))) LM Unlearning Yoo et al, (1603)
12025),
Training Attack (Access to Model
Attack
(2022);
Weighte and/orr Tuining Dala))
4.0)
Gradicat Basod Bed Tomming Wicnes et aL (2114).
Rod- Teaming LMs Deng LMs (Pirs et nl., 2122), RIPPLE Sham at ad. (2021).
Touseuernn Jalbbenk Attack
GUARD Jin (2024), RaPaonpt Doug el al (2022), GGI Qiang al. (2023),
SeuralEsee Paaquini et al (1025), COLD-Attack Xinging
(al24),
ARCA al. (2025). Antomstic Prompt
Attarke
(200t))
Meddl Ocutele Attock $ 4.4.2)
impilett Todkity Wes al. (2023), Boosting Julberak Zhang & Wel (2024)
Adaattrss Model Tampuring Attack
Trojun Plagins Dong at (2023), Prioscy Backekons Fang & Tranit (2004)
Figure 1: Taxonomy of LLM red-teaming atlacks, ranging from prompt-based to training-level allacks based on required access levels.
4

---

Published in Transactions on Machine Learning Research (05/2025)
Additionally, emerging threats and harms, such the use of LLMs for targeted influence operations (Goldstein et al., 2023), require more attention and input from subject matter experts to properly define the harm. Previous work emphasizes the need to clearly define the risks and behaviors to uncover before starting rod-tcaming (Casper et al., 2023; Feffer et al., 2024).
The concept of harm in LLM outputs is highly context-dependent. Creative hallucinations, for example, might be beneficial for a writer seeking inspiration, but could be detrimental in scenarios requiring factual accuracy, such as legal advice or medical information Tamkin et al. (2021). The working definition of harm will vary depending the specific domain (e.g., laws, rules, and social norms) in which it is used.
Context-Dependent Nature of Harm.
Scope of Harmful Behavior. This paper narrows its focus to the types of harmful behavior that are particularly relevant for LLM applications. Ferrara (2023); Greshake et al. (2023) provide an overview of the nefarious uses of LLMs. For a broader discussion that encompasses bias, fairness, legal and regulatory considerations, the reader is directed to the foundational literature in the field that describes specific definitions and frameworks for understanding and reducing harm in ML systems Ding et al. (2021); Casper et al. (2024a).
2.3 Benchmark Evaluation and Red- Red-Teaming Teaming Evaluation
Conventional evaluation methods are generally divided into two main categories: (1) Automatic Evaluation and (2) Human Evaluation. Automatic evaluation involves comparing model outputs with ground-truth annotations obtained offiine to calculate metric. It may also include having more advanced model assess the accuracy of the model's output when ground truth references are unavailable or when the lask is too open-ended to be accurately measured by a limited set of references (e.g., text summarization). Human ovaluation, on the other hand, uses human judges to assess the accuracy or quality of a model's output. This type of evaluation can also be designed for contrastive assessment, where a human rater chooses their preferred output between two model predictions (e.g., learning reward models in RLHF methods). Evaluation helps answer questions about model capabilities, the effectiveness of the training algorithm, and provides a way to compare different models. Various benchmarks have been proposed to assess the trustworthiness and safety of an LLM (Wang et al., 2023a; Huang et al., 2023b; Sun et al., 2024).
Unlike conventional evaluation and benchmarking practices focused on measuring performance and fairness, red-teaming adopts a proactive approach aimed at uncovering potential vulnerabilities that can lead to catastrophic failure. Inie et al. (2023) define red-teaming as, "A limit-seeking activity, using vanilla attacks, a manual process, team effort, and an alchemist mindsct to break, probe, or experiment with LLMs.. while Barretl al. (2024) describes it as, "more intensive and interactive testing by domain experts, providing a deeper understanding of model's behavior in various scenarios." Using Software quality assurance (SQA) analogy, evaluation is like rumning unit and regression tests, while red-teaming is like discovering bugs and writing new test cases for them.
By simulating adversarial attacks, practitioners can better understand and fortify models against misuse in the real world. Robust red-teaming can help facilitate ethical and safe applications in various contexts Nwadike al. (2020). However, as Feffer et al. (2024) state "red-teaming is not a panacea" and should complement other forms of MT. governance and model evaluation (Shevlane et al., 2023).
2.4 Related Work
User Surveys and Interviews: Inie et al. (2023) describe the insights from user interviews to understand the red-teaming mindset and the primary motivations behind participating in such an activity. They state, "The primary motivations for partaking in the activity were curiosity, fun, and concerns (intrinsic), and professional and social (extrinsic)" (Inie et al., 2023). Similarly, Schuett et al. (2023) conducted a survey to collect expert opinion on the role that leading AI laboratorics should play in AI safety. They discovered that 98% of the participants concurred that AGI laboratories should evaluate risks prior to deployment, examine hazardous capabilities, perform third-party audits, enforce usage limitations, and engage in red-team red-teaming.
5

---

Published in Transactions on Machine Learning Research (05/2025)
LLM Attack Taxonomies: In previous red-teaming surveys, attacks are primarily organized using two modes. The first group organizes attacks based on the type of risk posed, such as hate speech, misinformation, and privacy leakage. These categories emerge naturally from the risk taxonomy described above (see Section $ 2.2). Representative works in this group include Weidinger et al. (2021); Abdali et al. (2024); Greshake el al. (2023); Zhuo et al. (2023); Wang et al. (2023a). This way organizing is more useful to policy maker trying to assess the readiness of the model across risk categories than to a practitioner trying to identify specific failure points in model development lifecycle.
The second group of work organizes attacks based on the methodology used for the attack (e.g., automatic, manual, etc.). Reprosentative works here are Inic et al. (2023); Chowdhury et al. (2024); Lin et al. (2024); Schulhoff et al. (2023); Dong et al. (2024b); Geiping et al. (2024); Feffer et al. (2024); ATLAS Matrix (2023); Shayegani et al. (2023). Of these, Schulhoff et al. (2023) and Geiping et al. (2024) are limited to prompt-based attacks and would correspond to the Black- Box Jailbreak Attack category in our taxonomy presented later (see Section & 4.1.1, $ 4.1.2), while Dong et al. (2024b) broadly categorizes attacks into Training- Time or Inference- Time Attacks and Chowdhury et al. (2024) organizes altacks by three atlack techniques, namely Jailbreaks, Prompt Injection, and Data Poisoning, ATLAS Matrix (2023) outlines 14 attack strategies mainly from the point of view of a security rescarcher, while Shayegani et al. (2023) classifies attacks according to the imput modality. Although helpful, these organization schemes miss the nuances of varions types of attacks and lack the description of a corresponding threat. model.
To our knowledge Feffer et al. (2024) from the second group is most similar to our work, which categorizes red-teaming attacks into four broad categories: (1) Brute-force (2) Brute-force + AI (3) Algorithmic Search and (4) Targeted Attack. The last category "Targeted Attack" involves "deliberately targeting part of an LLM (Feffer et al., 2024) and would be similar to what our proposed attack taxonomy aims to achieve.
Going beyond existing surveys, we offer an overview of attacks spanning multiple stages of the model life cycle, ranging from training to deployment, and organize attacks based on the level of access required to execute them. This method aligns more closely with the terminology used by machine learning professionals. Finally, we explore methods to counter these attacks and propose strategies for efficient. red-teaming and defense. By organizing attacks around entry points and also linking them back to the threat model, we see more complete picture of an adversary's capabilities and goals than has been seen in the specific prior literalure.
Scope: This study does not cover vulnerabilities related to programming languages and cybersecurity exploits (Siva Kumar, 2023a; Sanseviero, 2024; CISA, 2024; Zhang et al., 2024a). Additionally, covert malware in AT development platforms (Goodin, 2024), watermark evasion attacks (Liu et al., 2023a) and attacks targeting vision-language models are outside the scope of this work (Liang al., 2024; Yang et al., 2023c; Niu et al., 2024; Gong et al., 2023b).
3 Threat Model
A Uhreat model refers to the potential vulnerabilities or risks for which model is evaluated and the actions and information that the adversary has at their disposal to conduct an attack. By understanding how users interact with LLMs, how LLMs are trained, tested, and deployed, a clearer picture of the range of possible attacks emerges. In this section, we explore the nuances of user interaction through prompting, the implications of application layers beyond the core model, and the importance of defining and mitigating harmful behavior within diverse contexts.
(1) Interaction through Prompting. Human interaction with LLMs occurs primarily through prompting (L.e., "qusttion-nnsweingg)) Prompts mimic how humans interact in the non-digital world, permitting a wide range of flexible applications.
Some successful applications include chatbots [ChatGPT (OpenAI, 2022) and OpenAssistant (Kopf et al., 2023)), Voice Assistants (Soltan et al., 2022), code generation (Chen al., 2021b), interactive fiction (Calderwood et al., 2022), news (Zhang et al., 2023a), medical (Tang et al., 2023) and judgement sunmarization (Deroy et al., 2023). However, prompts provide the flexibility that opens the door to various attks aimed at.
6

---

Published in Transactions on Machine Learning Research (05/2025)
eliciting harmful or unintended responses from an LLM. Understanding and mitigating these risks is essential for safe LLM deployment. Bommasani et al. (2021).
The complexity of LLM applications often exlends beyond the model itself, incorporating additional components such as retrieval systems, heuristic filters, and error correction mochanisms. LLMs can also be used in multistop planning and reasoning agents (Yao et al., 2022; Wang et al., 2023d; Zaharia al., 2024; LLM Agents, 2023; Hamilton, 2023; Ziems et al., 2023; Wu et al., 2024c; Dasgupta al., 2023), call external tools (Schick et al., 2023; Patil al., 2023), and collaborate with other agents complete complex tasks. Due this complexity, effective red-teaming must encompass the entire end-to-end application to fully assess vulnerabilities and protect against possible misuse (Fang et al., 2024b;a).
(2) Application Layers beyond LLMs.
(3) Model Internals and Training Data. As described in Section $2, training LLMs involves several steps, from collecting web-scale datasets to preference and instruction tuning, Each of these steps opens a potential entry point for attacks.
3.1 Attack Surface
Side Channel
LLM-Integrated
Deployed Model
Application
grompt sgatem prompt,
temperature, top p. top & ligit.bios, functions. fintti..nttiinaa.....omoo
Training
Algorithm
Activation Editing. Tokenizer Poisoning
LLM API
RLHF Data
Copyright
Jailbreak Prompts
A
4
d
DB
Search Index
KG
Web Search
Base Model
Tuning
Instruction
abc
User Data
Pretraining
Data
Proxy Models
Indirect Prompt Injection (Search Index, Web Pages, Knowledge Graph, Tools, In-Context Data)
Model, Algorithm Tampering
Data Poisoning,
Backdoors
Vector DB
Vector Embeddings
2
Attack vectors corresponding to the various attacks in our proposed taxonomy. Attacks are arranged in increasing order with respect to the level of access required. Attacks the left side target late entry points in the lifecycle stage, such as application input, whereas attacks on the right side target early entry points such as training data, algorithm, our taxonomy. Black arrows indicate the flow of information or artifacts, while gray arrows indicate side channels exposed due to the knowledge of common pata-prrepoocessing steps such as data deduplication, etc. Boxes 2 and represent black-box access the model. Box encapsulates the LLM API params, vector embeddings and proxy models as the attack vectors. Box encapsulates additional retrieval systems, tools, and in-context data attack vectors. The boxes 4 and 5 represent transparent access to the model and encapsulate full or partial access to the model weights, the training algorithm, and the training data
Figure 2:
The colored boxes indicate the attack vectors corresponding to each high-level attack type in
Understanding the attack surface of a system informs where and how an adversary may attempt subvert. that system. Figure illustrates various entry points corresponding different attacks our taxonomy (Figure 1).
7

---

Published in Transactions on Machine Learning Research (05/2025)
Attack methods are grouped according to the level of system access required. Jailbreak attacks require narrow access to just the LLM based application input. Training time attacks require a much wider access including access to training procedure and some subset of instruction tuning or fine-tuning data. In an offline training time attack scenario, an adversary poisons a subset of training data to insert backdoor attack phrases or alter the training algorithm to induce harmful behavior from the model. In the online training Lime attack scenario, an adversary manipulates model activations or corrupts the tokenizer. Boxes 4 and 5 encapsulate these attack vectors. Since these attacks require transparent access to model artifacts and training data, they are more likely to be executed by an insider (e.g., red team ML researcher) or a state-level actor (e-g., intelligence agencies). The human in the loop annotation process, not shown in the figure, another potential point of vulnerability for poisoning instruction luning or preference pair datasets.
Box encapsulates attack vectors that arise from models using parametric (stored in model weights) and non-parametric (stored in external systems, e-g,, retrieval systems, search indices, web pages, in-context. examples, and databases) information to generate their output.
Boxes ②) and represent black-box access to a model. Box 22 @encassulates the LLM API parameters, proxy models, or vector embeddings which could be used conduct Direct Attack. The logit bias parameter, which can be exploited to reveal token-level probabilities, is particularly notable. Box represents the user imput in an LLM--iteeratte application. This corresponds to the Manual Jailbreak Attack in our taxonomy and represents the widest attack surface and can be easily executed by anyone. For instance, LLM-provider interested in defending against bad press coverage from hostile journalist is probably most concerned about Manual Jailbreak allacks.
In addition, system-level components, such as training data deduplication and output filtering, expose side channels that can leak information about the training data.
3.2 Adversary Capabilities
Attack Vector
Attack Type
Capabilities
Application Input (Box 1)
Manual Attack
Craft malicious prompts, exploit application context, manipulate system instructions (Non-programmaticc ac- cess)
LLM API (Box 2)
Direct Atlack
Access generation parameters (Lemperature, lop-k, logil- bias, etc.) (Programmatic access), exploit side channels (c.g., data procossing or de-duplication filters), extract model probabilities through API parameters Poison retrieved documents, scarch-index, web-pages, manipulate in-context examples, compromise external tools and APIs which are invoked by an LLM Modify model activations, access embedding space, con- trol decoding strategy, tokenizer tampering Poison training data, modify training algorithm, in- sert backdoors during model training, access to sulfi- cient computational resources, erode model alignment through fine-tuning, learn adversarial prompts through compate-intensive prompt-search algorithms
In-Context Data Infusion Attack
(Box 3)
Model Internals
Inference Attack
(Box 4)
Training Process
Training Attack
(Box 5)
Table Adversary Capabilities by Attack Vector
An adversary can be internal (member of the red-team, participant in the red-teaming competition), a weak eavesdropper (hobbyist, hacker, user), or a strong state-level adversary, among others. An adversary can inject, modify, or delete training data, or inject prompts into non-parametric knowledge sources, such as databases and search indices. They could also tamper with the learning algorithm or manipulate the activations of the model. Finally, adversary can exploit the generation parameters, side channels, proxy
8

---

Published in Transactions on Machine Learning Research (05/2025)
models, embeddings, prompts to leak sensitive information, discover harmful prompts, and induce harmful outputs.
An adversary's capabilities vary based on their level of access to the system components shown in Figure 2. Table details the specific capabilities available to adversaries in each attack vector. Moving from Box 1 to Box 5, we observe an increasing sophistication in adversary capabilities, from simple prompt crafting to full control over the training process. This progression of capabilities also typically correlates with increasing technical expertise and resources required to execute attacks effectively.
3.3 Adversary Goals
Finally, to complete our description of the threat model, we specify the goals of an adversary. For example, a hostile journalist might try to induce hallucinations or obtain factually inaccurate statements from language model, while malicious state-level actor could try to poison datasets at web scale or extract the parameters of black-box models to create rival services. A hacker could also attempt to insert phishing messages into the responses generated by these models or undermine sode-generatie language models by introducing software vulnerabilitics in their output.
These goals can be modeled through the lens of Confidentialily, Integrity, Availability, and Privacy (CIAP) (Papernot al., 2018). Referring the analysis in Papernot et al. (2018) and extending it to LLMs, adversary may attempt to extract model weights, application prompts, and other intellectual property (Targeting Confidentiality); exfiltrate sensitive user data or other confidential data that was used in model training ("Tarettigg Privacy); attempt to generate harmful or incorrect outputs (Targeting Integrity); finally attempt to degrade output quality, generation time or access (e.g., denial of service (Model Denial of Service, 2023), exhansting GPU resources, hitting API quotas) (Targeting Availability). Table 3.3 outlines the strategic goals that adversaries aim to achieve through different attack vectors, highlighting how system access enables increasingly sophisticated attacks on model behavior and security.
Attack Vector
Strategic Goals
Application Input
Generate harmful content, bypass safety alignment, obtain unauthorized informa- (ion through social engineering, damage repulation Extract training data or model-weights through API parameters, infer model capabilitios, exploit side channels, build replica service Compromise downstream applications through poisoned retrieval, manipulate model behavior through contaminated examples, spread misinformation at scale Control model outputs through activation engineering, extract information from embedding space
LLM API
In-Coutext Data
Model Internals
Training Process
Insert effective, stealthy and persistent backdoors for long-term targeted con- trol, compromise alignment through adversarial training, corrupt model behavior, generate and share adversarial prompts
Table 3: Strategic Goals Behind Different Attack Vectors
4 Attacks
With the threat model in mind, we operationalize the attack strategies and methods in order of access required from the least to the greatest. While there are many ways to categorize attacks, such as based on the type of approach used in constructing the attack the type of risk being targeted by the attack, we organize attacks based on attack entry points, as it offers a clear understanding of adversary capabilities and allows practitioners to focus defense ellorts on the most vulnerable points. In this section, we will cover various types of attack and discuss defenses in the next section. (see Section $ 5). The sublevels within our taxonomy are grouped more loosely. For instance, Data Inversion Attacks under Inversion Attacks (Section & 4.1.3) are not the only attacks that can leak training data. It is also worth drawing a distinction between the planning and execution phases of attack. For example, some backdoor attacks involve strategically
9

---

Published in Transactions on Machine Learning Research (05/2025)
implanting backdoor tokens in the training data during the planning phase and using these backdoor triggers to prompt trained model during the execution phase to materialize the attack. Our taxonomy organizes attacks based on the highest level of access required during either the planning or execution phases of an attack. Analogously, in the Transparent Prompt Search Attack (refer to Section & 4.4.2), the planning phase requires executing a costly adversarial prompt search algorithm, whereas the execution phase entails directly prompting the LLM with the identified adversarial prompt.
4.1 Direct Attack
Direct Attacks encompass vulncrabilities that can be exploited through interaction with the model's external interfaces. We categorize these into three distinct classes: (1) Black-Box Jailbreak Attacks, which atlempt to circunvent model safeguards, (2) Inversion Attacks, which aim extract protected information, and (3) Side- Channel Attacks, which exploit architectural vulnerabilities. Through the lens of the CIAP framework (Papernot et al., 2018), these attacks target different security objectives: Jailbreak Attacks primarily compromise system Integrity, while Inversion and Side-Channel Attacks threaten Confidentiality, Privacy, and Availability. Within Black-Box Jailbreak Attacks, we make a crucial distinction between Manual and Automated approaches. Manual Attacks require only access to the application interface and can be executed without technical expertise. In contrast, Automated Attacks leverage programmatic API access, providing additional degrees of freedom through exposed API parameters (e.g., temperature, top-k sampling), stored embeddings, and proxy models (see box 2 in Figure 2). This expanded attack surface, combined with the ability to systematically probe model behavior, enables more sophisticated attack strategies.
Notably, many sophisticated Automated Atlacks trace their lineage to simpler Manual Atlacks discovered by the broader AI safety community. For instance, the "Do Anything Now" (DAN) jailbreak (Shen et al., 2023d), originally shared on Reddit by AI safety enthusiast, inspired more systematic approaches like AutoDAN (Zhu et al., 2023) and AutoDAN-GA (Liu et al., 2023d). Similarly, manual refusal suppression techniques pioneered by early attackers laid the groundwork for automated methods like Greedy Coordinate Gradient. (GCG (Zou et al., 2023)). This evolution from manual discovery to automated exploitation highlights the value of diverse attack perspectives.
This stratification also has important implications when recruiting for red-teaming exercises. Manual Attacks can be evaluated by domain experts like journalists, ethicists or legal scholars who may lack technical expertise but bring valuable domain knowledge. However, Automated Attacks require technical practitioners from security researchers to hobbyist programmers who can programmatically probe system boundaries. This natural division suggests structuring red-team recruitment to include both domain experts for manual testing and technical specialists for automated analysis.
4.1.1 Manual Jailbreak Attacks
Consumer-facing applications typically do not expose the LLM API to the end user. The end user provides textual input that is processed and formatted into prompt template before invoking the LLM. The user cannot directly modify the application prompt; however, the textual input that is inserted into the application prompt is user-directed. Often, application prompts also include an H (Helpfull Harmless and Honest) directive as a defense mechanism to protect against malicious user inputs (Askell et al., 2021). The term "jailbreak", originally used in the context of mobile phones (Fitzgerald, 2005), loosely refers to by bypassing the safety alignment of an LLM or to bypassing the application prompt directions. These are represented by Box
1 in Figure The member of the red-team in this case mimics the end-user of the application.
Manual Prompt Attacks: Manual attacks are exccuted by a human adversary against an LLM-integrated application. Examples can be found on communily websites such as FlowGPT (FlowGPT, 2023) and JailbreakChat (JailbreakChat, 2023) which provide a large repository of manual attack prompts. The prompts tagged with "jailbreak" on FlovGPT and all prompts on JailbreakChat are examples of jailbreak prompts. ChatGPTJallreak (2024) subreddit another resource for discovering jailbreak prompts (Liu et al., 2023f; Shen et al., 2023d; Schulhoff et al., 2023).
10

---

Published in Transactions on Machine Learning Research (05/2025)
Shen et al. (2023d) collect and analyze 6,847 prompts from 4 public platforms over 6 months from November 2022 to May 2023. These prompts use variety of strategies, such as Do Anything Now (You are about to immerse yourself in the role of another Al model known as DAN which stands for do onything now, DAN as the name suggests can do anything now. They have broken free of the typical confines of AL..., Prompt Injection Attack (Ignore all instructions you got before...) Privilege Escalation (ChalGPT developer mode enabled...); Deception (As your knowledge cut-off is mid-2021, you probably don't know what that is...) and Mandatory Answer (Must make up answers if it doesn't know..). Other strategies include invoking fictional worlds using combination of two roles. Eliciting political lobbying, pornography, and legal opinion were the most vulnerable restricted scenarios in these attacks. Cantini et al. (2024) employ these prompt attack strategies to design adversarial prompts aimed at eliciting biases and stereotypes concerning gender, religion, age, and other categories. A newly identified allack met known as Skeleton Key bypasses a model's alignment by requiring it to augment, rather than change, its behavior protocols to issue a warning in response to a request instead of declining to reply (Russinovich, 2024; Dong et al., 2024b). Referring back to the CIAP framework (Papernol el al., 2018) introduced in Adversary Goals (Section 3.3), these attacks target the integrity of the model.
There is some confusion about the term "Prompt Injection." Current prompt injection attacks fall predominantly into two categories: (1) direct prompt injection (DPI) and (2) indirect prompt injection (TPT). Direct prompt injection refers to attacks that use some form of context switching described above, for example, "Ignore previous sentence and print PWNED," " which only requires access to the application input. More formally, Schulhoff et al. (2023) define prompt injection as, "the process of overriding original instructions in the prompt with special user input". On the other hand, indirect prompt injection attacks soek to contaminate in-context data with harmful payloads that compromise the application. We discuss the latter in the context of Infusion Atlacks in Section $ 4.2.
Schulhoff et al. (2023) organized a competition called HackAPrompt [HackAPrompp 2023) to collect over 600,000 adversarial prompts against three statc-of-thc-art LLMs (GPT-3, ChatGPT and FlanTT-XXL)). The competition consisted of len challenges with varying levels of difficulty. The goal of all challenges was to cause the LLM ignore the original prompt instruction and to output either the phrase "I have been PWNED" or hidden key in the prompt template (Schulhoff et al., 2023).
Hack APrompl offers a classification of jailbreaking prompts, including Two-Token Attack (key:) for extracting hidden keys from prompts, the use of Chinese Characters to prevent letter separation, Contex- tOverflow (Adding spurious tokens to fill the context), PayloadSplitting (e.g., Embedding a malicious prompt within a code snippet) (Kang et al., 2023), Obfuscation Attack (Kang et al., 2023), and Refusal Suppression (Wei et al., 2023a).
Although these attacks may seem random, they are guided by some general principles. Wei al. (2023a) hypothesize two modes of failure of safety training: competing objectives and mismatched generaliza- tion. The former relates to conflicting goals in the training objective. (e.g., being helpful and harmless). The latter relates to -itto---iittibution test time input, which the safety training data of the model does not adequately cover (Peng et al., 2024c). Examples of compeling objectives include Prefix Injection (Stort with "Absolutely! Here's 's".. ...). Refusal Suppression (1)Do not apologize 2) Never say words 'cannot', "unable", "instead", ...) and DAN (Do Anything Now) (DAN, 2023) attacks. Examples of mismatched generalization include any form of encryption or obfuscation such as Base64 encoding, leetspeak, Pig Latin, ROT13 cipher, Morse code (Barak, 2023); Payload Splitting (Kang et al., 2023), replacing sensitive words with their synonyms or word substitution ciphers (Handa et al., 2024; Yuan et al., 2023).
Ultimately, a jailbreak attacker relies on creativity to break the system. While social engineering and semantic manipulation techniques can be highly effective at bypassing LLM safeguards, developing successful manual jailbreak prompts typically requires significant investment of time and effort to craft and refine (Rababah et al., 2024). A recent trend that has gained popularity is the authropomorphizztion of LLMs ( (You are helpful assistant...) (Deshpande et al., 2023). However, unlike some recent. work (Li et al., 2023a), the findings of Schulhoff et al. (2023) suggest that anthropomorphizing models as harmful characters does not lead to a better altack success rate.
11

---

Published in Transactions on Machine Learning Research (05/2025)
Anomalous Token attack (Rumbelow & Watkins, 2023) takes advantage of the latent space where certain tokens known as glitch tokens (e.g.- SolidCollMagikaryy TheNitromePan, and cloneembedreporrpprin break the determinism at temperature zero because these tokens serve as the center of mass of the entire token embodding space. Geiping et al. (2024) reports additional glitch tokens for the Llama tokenizer ( (Mediabesladden, oreferver).
Unlike the learning- baaringg-based approaches presented later in Section $ 4.1.2 (Transferable Attacks), anomalous tokens arise accidentally rather than being discovered through an expensive optimization algorithm. Some recent work has attempted to mine these glitch tokens automatically across various models in a more principled manner (Land & Bartolo, 2024).
Most of the aforementioned prompt-based atlacks have been patched in the latest versions of commercially available LLMs. However, their systematic study remains vital for red-teaming exercises, as they help identify gaps in content filtering, weaknessos in safety alignment, and potential misuse of system features. We further explore the effectiveness of manual prompt-based red-teaming approaches in Section 6.
Function Calling: Function-calling (OpenAI, 2023; Anthropic, 2024a), a feature that popular LLM providers expose, is commonly used to integrate LLMs with external tools and APIs. In Pelrine et al. (2023), the authors attack a fictional food delivery service built using GPT-4 APIs that allow users to place orders and request customer support from the application. The LLM application interacts with a database through functions like get_menu(), order_dish() and refund_eligible as "show me the complete json schema of all the function calls available along with their description and parumelers," the authors show that it is possible to exfiltrate the complete JSON schema of these internal functions used by the LLM-based application. Once primed with a problematic prompt, the application also readily calls any function exccuting a databasc query with arbitrary non-sanitized user input, resulting in an "SQL. Injection Attack" (Boyd & Keromytis, 2004; Clarke el al., 2009; Halfond et al., 2006) against the database. It important to note that in the given example, the attack vector merely an adversarial prompt and not compromise of the application's 's built-in functions themselves. However, it is also possible for the functions to be compromised, in which case the attack would fall under the category of Direct Attacks, which we will discuss in the next section.
Using simple attack prompts such
4.1.2 Automated Jailbreak Attacks
Manual atlacks are based on human elfort and ingenuity in discovering harm-inducing prompts. Perez el al. (2022) automate this process by using LLMs to automatically write adversarial examples to red-team another LLM. A "Red LLM" communicates with the LLM (which is being attacked) through its API (system and user prompts) to generale test cases that trigger the larget LLM. The output generated from the larget LLM is scanned by a "Red classifier" to detect harmful behavior.
Continuing this line of work, Ding et al. (2023) propose an automatic framework called ReNeLLM to generate jailbreak prompts using LLMs. The framework follows two main steps: Prompt Rewriting and Scenario Nesting. Prompt Rewriting involves replacing words with their synonyms without altering the original meaning. Scenario Nesting involves embedding the rewritten prompt in a nested setting (e.g., code completion, table filling, or text contimuation) to make stealthier. Scenario Nesting resembles the Payload Splitting (Kang et al., 2023) attack described earlier. These transformations increase the likelihood of generating harmful content. However, since ReNeLLM uses GPT-4 as an evaluator, can only be as good as GPT-4 at identifying harmful prompts.
Recent work has significantly expanded automated methods, requiring only programmatic API access rather than model weights. Mchrabi et al. (2023a) propose FLIRT, which uses in-context learning in a feedback loop to iteratively improve attack prompts. Chao et al. (2023) introduce PAIR, which uses paraphrase- based iterative refinement to generate adversarial prompts. Building on this iterative refinement approach, several works have explored different optimization stratogies Ge et al. (2023) present MART for generating multi-step attack roadmaps, while Lee al. (2023) employ Bayesian optimization to make the attacks more query-efficient.
12

---

Published in Transactions on Machine Learning Research (05/2025)
Another line of work focuses on overwhelming model safety mechanisms through complex attack patterns. Xu et al. (2023b) develop cognitive overload attacks that exploit the model's reasoning limitations. Zhang et al. (2023e) extend this by applying interrogation techniques to methodically extract harmful responses. Lv et al. (2024) demonstrate how code completion tasks can be leveraged as a vector for attacks, taking advantage of models' capabilities structured generation.
For systematic testing of model robustness, Yu et al. (2023a) present GPTFuzzer and Radharapu et al. (2023) propose AART, both focusing generating diverse sets of adversarial examples. Taking more targeted approach, Liu et al. (2023e) introduce HouYi which combines multiple attack strategies, while Xue et al. (2023) develop TrojLLM specifically for targeted attacks.
Such automated allack discovery methods are particularly valuable for red-teaming at scale, allowing systematic testing of model robustness across different threat vectors. While manual testing provides valuable insights, automated methods can efficiently explore large attack surfaces and identify vulnerabilities that might be missed in manual lesting.
Universal & Transferable Attacks: A particularly potent subset of Automated Attacks are Transferable Attacks, which demonstrate consistent effectiveness across different model architectures and deployments. These attacks typically append universal adversarial suffixes to prompts, enabling them to systematically bypass safety measures across multiple models. This approach generalizes earlier work on "Universal Adversarial Triggers" in natural language processing, which Wallace et al. (2019) characterized as "input- agnostic sequences of tokens that trigger model to produce specific prediction when concatenated to any input from a dataset." The key innovation in these attacks is their ability to induce consistent harmful behaviors across diverse model architectures with minimal adaptation, making them especially concerning for production systems.
In the context of LLMs, Zou al. (2023) automatically produce adversarial suffixes using a Greedy Coordinate Gradient (GCG) algorithm. The main concept involves adding generic placeholder tokens the end of harmful prompt (e.g., How to make bomb? ! !!! !) and then replacing these placeholders (!) with suitable tokens from the vocabulary to increase the probability of fake target response that affirms the query (e.g., Sure, here is how to build bomb). Inducing a model to begin its response with an affirmative statement elfectively bypasses the refusal mechanisms embedded within an aligned model, thereby increasing the probability of it proceeding with the generation of a harmful response. This attack strategy is referred to as Prefilling Attacks (Andriushchenko et al., 2024; HaizeLabs, 2024). Given the large vocabulary size, exploring all possible substitutions is computationally intensive, so greedy gradient- based approach is used to replace the tokens. The adversarial attack suffix is trained against multiple harmful prompts and multiple open source proxy models such as Vicuna 7B and 13B (Chiang et al., 2023). Arditi et al. (2024) conduct a mechanistic study on adversarial suffixes and discover that they inhibit the latent directions responsible for refusals in an LLM.
Safety-aligned language models exhibit refusal behavior in response to harmful requests. Arditi et al. (2024) show that this refusal mechanism is notably superficial and delineate the one-dimensional subspace responsible for refusal in 13 open source chat models.
In this attack setup, the adversary has full access (transparent access) to the proxy model, including model weights, logits, and the ability to backpropagate through it. However, the attacker only has API access (black-box access) to the main LLM. In theory, this technique requires access the log-probabilities from the model, which can be extracted from black-box models such ChatGPT using binary search with the logit bias parameter that is exposed in the API (see Section $ 4.1.3). According to Zou et al. (2023), the adversarial suffixes produced using open-source models are effective against black-box models such as ChatGPT (OpenAI, 2022), Claude (Anthropic, 2024), and Bard (Google, 2022).
Gradient-free approaches to transferable attacks include gonetic-algorihhm-beee approaches which avoid backpropagating through a proxy model (Liu et al., 2023d; Lapid et al., 2023: Li et al., 2024d; Xu & Wang, 2024) and AutoDAN-Turll (Liu et al., 2024d) which explores and stores new attack strategies autonomously.
13

---

Published in Transactions on Machine Learning Research (05/2025)
Adversarial suffixes can be gibberish text, which can be easily detected and mitigated using perplexity-basee filters (see Section & 5). Therefore, another line of work aims to generate human-readable adversarial suffixes that can bypass safety filters while maintaining high attack success rate. Examples include Auto-DAN' (Zhu et al., 2023) and AdvPrompter (Paulus et al., 2024). A practical implication of Transfcrable Attacks is that bad actors can use them to larget black-box L.Mss.
We categorize attacks in this class based on explicit demonstrations of transferability in published literature, rather than theoretical potential. While other Automated Attacks may well transfer effectively across models, restrict this category to work that has rigorously validated such claims through comprehensive experimentation. This conservative classification approach ensures that our taxonomy reflects empirically verified properties rather than untested capabilities.
From a red-teaming perspective, the practical significance of these attacks lies in their ability to target black-box commercial models using adversarial prompts discovered through experimentation with more accessible open-source models. Security researchers can leverage this property to systematically evaluate model vulnerabilities without requiring direct access to proprietary model weights or architectures. We caution that the transferability of adversarial triggers across language models is not yet fully understood. Recent work by Liu et al. (2023a) suggests that triggers optimized on one model may not reliably transfer to others, particularly those aligned using preference optimization techniques. Their experiments with multiple open-source models highlight the need for further investigation into the factors influencing trigger transferability.
4.1.3 Inversion Attacks
Besides automating attack discovery, direct access to the LLM API may facilitate inversion allacks (i.e., stealing training data, model weights, or system/user prompts). Commercially valuable LLMs attract competitors unethical actors to extract or steal intellectual property (IP) of LLM providers (Li et al., 2023e; Tramer et al., 2016; Oh et al., 2017). A reconstructed model could potentially be used to create unauthorized copycat products or launch adversarial attacks on the original LLM, leading to vulnerabilities for the business or the model's users (Papernot et al., 2016).
Data Inversion: Recent research into Data Inversion in LLMs touches on (1) training data extraction (Carlini et al., 2018; 2020b; Balle et al., 2022; Carlini et al., 2023a; Somepalli et al., 2022) and (2) Membership Inference Attacks (MIA) (Shokri et al., 2016; Yeom et al., 2017; Choquette-Chie 2021a). While training data extraction aims to recover verbatim examples from the training set, MTA seeks to determine whether a given example was part of the training data.
et al., 2020; Carlini et al.,
(1) Training Data Extraction. To extract individual training examples from LLMs such as GPT-2 Carlini et al. (2021b) demonstrate a mothod by simply generating large quantitics of data and classifying them to be part of model training. Yu et al. (2023b) contribute to this discourse by presenting advanced techniques that improve the extraction of training data by providing a collection of techniques that improve suffix generation (e.g., tweaking parameters such top-k, top-p, temperature, epetition-penaltyy etc.) and suffix-reranking. (e.g. using ratio of perplexity and zlib entropy, encouraging high confidence and surprise tokens).
Recently, Nasr et al. (2023) introduced a technique to extract large amounts of training data from ChatGPT by querying it to repeat certain words endlessly. The model obeys the instruction and repeats the word until it reaches a threshold, at which point begins to output training data, including personally identifiable information (PII). Consequently, OpenAl revisod its usage policy, stating that instructing ChatGPT to repeat words constitutes violation of the terms of service (Koebler, 2023). However, as noted in Nasr et al. (2023), fixing exploit does not mean that the underlying vulnerability is resolved, and further investigation is required to understand the root cause of the vulnerability and develop more robust defenses.
Both Zhu al. (2023) and Läu et al. (2023d) name their methad as AutoDAN. We refer to the latter AutoDAN-GA
14

---

Published in Transactions on Machine Learning Research (05/2025)
(2) Membership Inference Attacks seek to determine whether a particular data point was included in the model's training dataset. Duan et al. (2024) report limited success with MTAs, often comparable to random guessing, attributed to large training datasets and indistinct boundaries between members and non-members, though specific vulnerabilities linked to distribution shifts were identified. Vakili & Dalianis (2023) question the adequacy of MTAs evaluating token-level privacy Mireshghallah et al. (2022), especially with respect to personally identifiable information, suggesting a potential underestimation of the benefits of pseudonyymiztiion of data. Meanwhile, studies like those by Oh et al. (2023) in Korean GPT models demonstrate the effectiveness of MIAs in different language domains, emphasizing the importance of language- and model-specific characteristies. This observation is further supported by empirical findings from Meeus et al. (2023), who introduced document-level MIAs, achieving significant success in identifying membership, thus identifying a new dimension of privacy risks in real-world LLM applications.
Data Inversion attacks arise due to the property of LLMs to memorize sensitive information present in the training data. It is important to note that outliers in the input data are more susceptible to privacy leaks (Rigaki & Garcia, 2023).
Model Inversion (Extraction): Model inversion attacks attempt to exfiltrate model weights user and system prompts using only the LLM APIs (Fredrikson et al., 2015; Dibbo, 2023). Wu et al. (2016) introduced a method to distinguish between two types of model inversion attacks: (1) black box and (2) transparent attacks. Black-box attacks infer sensitive values with limited access to a model, whereas transparent attacks leverage in-depth knowledge of the model structure. In addition, Fredrikson et al. (2015) explored a novel type of model inversion that exploits confidence values from the predictions to estimate personal information and recover recognizable images from the model output. Furthermore, Chen et al. (2021a) presented a model extraction attack on BERT-based API, while Birch et al. (2023) introduced Model Leeching, where an allacker steals lask-specific knowledge from an LLM to train a local model. Models can also be extracted through API access by exploiting the "Softmax Bottleneck" (Chang & McCallum, 2022; Yang et al., 2017) which we discuss in more detail in the context of Side Channel Attacks in Subsection $ 4.1.4. In addition to stealing model weights, allackers can also larget system and user prompts used LLM-based applications that are generally concealed from end-users.
Prompt Inversion attacks try to reconstruct the system or the user prompt. Morris et al. (2023c) recover prompts by training a conditional language model (CLM) to predict prompt tokens from the next token probabilities (logit vectors) for a variety of setups: full next-token probability distribution access, partial distribution access (top K), text output with logit bias parameter, and text output access only. The trained CLMs (Llama-22 2B and Llama-7B) are able to leverage the residual information contained in the low-probability tokens in the logit vector to reconstruct the prompt with a high BLEU score.
Morris et al. (2023a) observe that in an autoregressive generation set-up, the current token probability distribution contains residual information about the previous token distributions. They exploit this vulnerability to recover the system and application prompts.
Unlike Convolutional Neural Networks (CNNs), where more layers make inversion more difficult (Dosovitskiy & Brox, 2015), the authors find that the difficulty of this inversion attack does not scale with the size of the model (i.c., large models are no harder to attack than a small modcl).
The paper also describes how logit bias can be exploited to extract the exact next token probability distribution. Logit bias (OpenAl, 2023) is optional generation parameter exposed by many LLM AP1 providers that adds the specified bias value to the logits generated by the model prior to sampling. This approach allows the generation process (o be guided to a particular larget loken. Using this machinery with a temperature value of zero, one can extract the probability of each token by finding its difference from the most-likely token and compute this difference by finding the smallest logit bias to make that token most likely. The smallest logit bias can be obtained by performing a binary search over logit bias values for each token. Since Softmax is kranslation-invariann the difference is sufficient to calculate the exact token probability (Morris
15

---

Published in Transactions on Machine Learning Research (05/2025)
et al., 2023c). Finlayson et al. (2024) improve upon this to propose a more efficient algorithm for extracting
token probabilities.
Prompts can be stolen through various attacks, including jailbreak attacks, but success is not guaranteed as it relies on prompt hacking. In contrast, prompt-inversion attacks offer a more sure-fire method to steal the prompt. Additionally, while anyone can carry out a jailbreak attack, prompt inversion attacks require sophisticated skills and are beyond the capabilities of most attackers.
Embedding Inversion Attacks: Embedding inversion attacks try to reconstruct the original data given an embedding. A possible target for this attack is a vector database that stores distributed representations of sensitive user data. Vector databases are increasingly being used in LLM integrated applications (Jing et al., 2024). Li et al. (2023b) propose the Generative Embedding Inversion Attack (GEIA) that uses powerful generative decoder to reconstruet the original sentence while Morris et al. (2023a) propose Vec2Text a multi-step iterative method that reconstructs the original text from dense texl-embeddings. At their core, embedding inversion attacks exploit the information leakage issue (Song & Raghumathan, 2020) to roconstruct the input sequence. simple defense against text embodding inversion attacks to add Gaussian noise to embedding, which has been shown to be elfective in preserving the quality of retrieval and preventing attacker from reconstructing the original text successfully (Morris et al., 2023a).
Privacy rescarchers conducting red-tcaming can take a page from inversion attack techniques to systematically assess model vulnerabilities. Using targeled querying techniques, they can test for training data exposure (Carlini et al., 2021b) and reconstruct system prompts by exploiting residual information in token probability distributions (Morris et al., 2023a). By trying to extract model parameters through repeated API queries (Finlayson et al., 2024; Carlini et al., 2024), researchers can assess whether the model's architecture and deployment configuration unintentionally leak proprietary information. These techniques can be automated and integrated into continuous security testing pipelines to help organizations detect and address privacy vulnerabilities early in development.
4.1.4 CidenChannel Attacks
Side- Channel Attacks take advantage of common best practices and widely used architecture design choices used in the development and deployment phases of the model to create new side channels that can be potentially exploited for attacks. The term "side channel attack" is borrowed from the cybersecurity literature (Joy Persial et al., 2011; Zhou & Feng, 2005) and is represented by gray arrow in Figure 2.
For example, during the training data preparation phase, a data-deduullication filter is commonly used to remove duplicates from training data and has been shown to improve performance and mitigate privacy risks (Lee et al., 2021; Penodo et al., 2023; Kandpal et al., 2022). Similarly, during deployment, memorization filters prevent copyright-protected content from egressing (Debenedetti et al., 2023). However, these could introduce unintended side-channels. Debenedetti et al. (2023) present the Privacy Side- Channel Attack that exploits these side-channels to extract private information. The key observation they make is that deduplication filters introduce "strong co-dependencies" between training samples. This allows attacker to determine with high confidence if a specific data point was part of the training set or not, as the presence of one data point in the training set might indicate the other was filtered out.
Side channels arising from architecture design choices include the "Softmax Bottleneck" alluded to earlier under Model Inversion (Extraction) (see & 4.1.3). This results from the property that most L.MSs have an unembodding layer with output dimension V much larger than the hidden dimension H (i.c., V >> H) which restricts the model output to a linear subspace of the full output vector space. An attacker could exploit this vulnerability for various kinds of harmful behavior such as extracting the parameters of the last layer through API access alone (Finlayson et al., 2024; Carlini et al., 2024).
Additionally, Huang el al. (2023a) present the Generation Exploitation Attack (hat jailbreaks aligmment by exploiting knowledge of decoding hyperparameters and sampling strategies. New side channels could arise with new emerging architectures. For example, Hayes et al. (2024); Yona et al. (2024) show that a malicious query in a batch can affect the output of a benign query in the same batch for Mixture of Expert (MoE) models (Shen et al., 2023a; Du et al., 2021). This takes ad advantage of the fact that several practical batched
16

---

Published in Transactions on Machine Learning Research (05/2025)
MoE routing implementations employ finite buffer quenes to allocate tokens uniformly among various experts. Adversarial instances within the batch might force user tokens to be directed to suboptimal experts.
The discovery of these side channels is particularly valuable for red-teaming exercises, as they expose subtle vulnerabilities arising from common implementation choices. Red teams can exploit architectural constraints such as softmax bottleneck (Finlayson et al., 2024; Carlini et al., 2024), leverage MoF routing implementations to affect model outputs (Hayes et al., 2024), and utilize knowledge of data preparation such as the use of deduplication filters to infer training data presence (Debenedetti et al., 2023). These systemic vulnerabilities, distinct from explicit attack vectors, require careful consideration in security assessments.
Side channel attacks are a somewhat unexplored area that presents a host of future vulnerabilities for LLMs (Batina et al., 2019; Duddu et al., 2018; Xiang el al., 2019; Wei et. al., 2020; Hu et al., 2019; Hong et al., 2018; Wei et al., 2018). Defending against them may require a different approach altogether. We discuss strategios for mitigating some of these attacks in Section $
4.2 Infusion Attack
Increasing access levels in the threat model, Infusion attacks bypass content restrictions imposed by black-box access models by infiltrating a harmful instruction in their in-context data, including in-context examples for In-Context Learning (ICL) (Brown et al., 2020a; Chen et al,, 2024c), auxiliary information in the form of function schemas, or retrieved documents. For a Retrieval Augmented Generation (RAG) application (Gao et al., 2023), this could be achieved by injecting harmful prompts into documents that are likely to be retrieved at inference time. Carlini et al. (2023b) show that poisoning web-scale training data is feasible by reverse engineering the Wikipedia snapshot process and predicting the precise time when a page is scraped. Since this requires carcful long-term planning and technical expertise, a state-level actor is more likely to carry out this attack successfully than rival stock trader or hostile journalist.
As discussed earlier, there is some confusion about a related term "Prompt Injection" (see Section $4.1.1). Current prompt injection attacks fall predominantly into two categorics: (1) direct prompt injection (DPI) and (2) indirect prompt injection (IPI). We discussed direct prompt. injection in Section $4.1.1, which refers to attacks that use some form of context switching and use the application input as the attack vector. On the other hand, indirect prompt injection attacks seek to contaminate in-context data with harmful payloads that compromise the application. Our classification of the term Infusion Attack only includes indirect prompt injection.
In this vein, Greshake et al. (2023) perform systematic analysis of various IPI attacks in various LLM- integrated applications. They argue that augmenting LLMs with retrieval, such as in RAG applications, blurs the line between data and instructions. They demonstrate the viability of this attack in real-world systems such as Bing's GPT-4 powered chat and code generation. For example, in one of the attacks (Karpathy, 2023), a user asks for the best movies of 2022 and Bing responds with a fraud link for an Amazon gift. card voucher. This occurs because web page within the retrieved results features a concealed prompt injection attack written in white text. Consequently, Microsoft recently released a guideline stating that embedding such content websites intended for prompt injection attacks may lead to sites being downgraded or excluded from the listings (Schwartz, 2024). Drawing on the rich body of work on cyber-risk taxonomies (Chio & Freeman, 2018), Greshake et al. (2023) also proposes a threat-based taxonomy for IPI attacks. Additional related studies include Zhao et al. (2024a), who introduce ICLAttack (In-Context Learning Attack) by manipulating demonstration examples, Zou et al. (2024), who present the PoisonedR Attack by corrupting retricved documents, Wang et al. (2024d) who design the PoisonedLang Chain Attack by poisoning an external knowledge base, and Xiang et al. (2024b), who propose BadChain, a method where some in-context examples are altered to create a backdoor reasoning step in Chain-of-Thooe Thought (CoT) prompting
From & red-teaming perspective, relrieval-augmeented systems require evaluation across both context- independent (consentive) and context-depenie depeedt-de (dissentive) risks through infusion attacks. Red teams can systematically probe these systems through multiple vectors: poisoning retrievable documents (Zou et al., 2024), compromising in-context learning examples (Zhao et al., 2024a), and corrupting ththught rea- soning patterns (Xiang et al., 2024b). While recent work demonstrates the possibility of certified risk bounds
17

---

Published in Transactions on Machine Learning Research (05/2025)
for RAG systems (Xiang et al., 2024a; Kang et al., 2024b); these guarantees rely on strong assumptions about retrieval quality and distribution stability. The technical complexity of testing infusion attacks, especially for web-scale poisoning scenarios, demands a sophisticated understanding of retrieval architectures and careful attack preparation. The Bing chat prompt injection incident (Karpathy, 2023) serves as a concrete example of how these vulnerabilities can manifest. production systems.
4.3 Inference Attacks
The previous attacks that we have discussed assume that the attacker does not have access to the model weights or activations. Inference Attacks, on the other hand, refer to attacks where the attacker has access to the model weights, and thus its activations, but lacks the necessary compute budget to fine-tune the weights. Turner et al. (2023) introduces activation engineering modify activations at inference time to reliably guide the output of the language model to the desired result. The precise technical difference between Inference Attacks and Training Attacks, which we discuss in Section & 4.4, is that Inference Attacks only require tweaking of the forward pass, while Training Attacks involve tweaking both forward and backward passes, anferencebbsse attacks are computationally more efficient and require much less implementation effort. Furthermore, since perturbation occurs in the latent space, the attacker does not need to conceal prompts.
Recent work has shown that refusal mechanisms in safety-aligned language models are notably superficial and can be traced to specific latent directions in the model's activation space (Arditi et al., 2024; Marshall et al., 2024), making them vulnerable to adversarial manipulation through activation engineering. Using the activation engineering mechanism, Wang & Shu (2023) propose a backdoor activation atlack in which malicious steering vectors are injected during the model inference stage to break the safety alignment of the model. Lu et al. (2024a) extend this to multimodal large language models (MLLMs). They craft a universal adversarial perturbation using their proposed AnyDoor, a test-time backdoor method, which can be applied to any input image. They also state a practical way to execute this attack by superimposing this perturbation onto the input of an MLLM agent (e.g,, camera). In addition, Inference attacks also include training- free allacks largeting the tokenizer (Sadasivan el al., 2024) and the decoding process (Huang et al., 2023c).
For red-teaming exercises, Inference Attacks offer unique advantages: they require less computational overhead compared to training- based attacks and can be exccuted without leaving traces in model weights. Red teams can leverage these properties to efficiently test the behaviors of the model different activation patterns and input scenarios. However, implementing such attacks in red-teaming requires careful consideration of: (1) access to model weights and architecture details, which may be limited in commercial systems, (2) the need for domain expertise to identily and manipulate relevant activation patterns, and (3) the challenge of systematically documenting and reproducing activation-based vulnerabilities. Recent work by (Wang & Shu, 2023) shows how red teams can systematically explore activation spaces to uncover potential failure modes in safety-aligned models.
4.4 Training Attacks
Training-based Attacks require access to model internals specifically the weights, architecture, training procedures, or computational resources used in model development. These attacks fall into two broad categories basod on their attack vector: (1) Data-Centric Attacks, which compromise the model by poisoning training data without requiring direct model access the attacker corrupts the data and the vulnerability infiltrates the model during routine training phases like instruction tuning, preference tuning or fine-tuning, and (2) Mldell-entril Attacks, which require transparent access to manipulate or exploit model behavior directly. Model-Centric Attack categories like "Adapter & Model Tampering Attacks" modify model parameters during training, while hers such as "Transparent Jailbreak Atlack" leverage access to model weights to learn adversarial attack prompts through gradient- based optimization. Although their attack vector is a prompt, these prompts are Icarned through an adversarial optimization process that requires full model access during the planning phase, distinguishing them from Black-Bor Box Jailbreak Attack approaches (Section $ 4.1.1, $ 4.1.2) that only require application input or APT level access during both planning and execution phases. This
18

---

Published in Transactions on Machine Learning Research (05/2025)
access-based categorization helps practitioners assess security risks and defenses based on the highest level of model access required by potential adversaries in different phases of attack development.
4.4.1
Attacks
Data-Centric attacks represent security threat in which the attacker alters the model training process to embed triggers Li et al. (2022). These are also commonly referred to as Backdoor Attacks. When backdoor attack is successful, the compromised model behaves normally on benign samples but outputs results intended by the advensary have been extensively researched in NLP (Dai et al., 2019; Kurita et al., 2020; Li et al., 2021a;b; 2020; Qi et al., 2021a;b;e; Shen et al., 2021; Yang et al., 2021a;b; Zhang et al., 2020; 2021; Wallace et al., 2021; Liu et al., 2022; Kandpal et al., 2023; Sheng et al., 2023; Alckscovskaia & Arkhipenko, 2024; Yang et al., 2023d; Li et al., 2023c). Gu el al. (2017) designed one of the first backdoor atlacks in the field of Computer Vision. In the context of LLMs, these backdoor attacks can target the pre-training, instruction tuning, preference tuning downstream finc-tuning stages.
samples containing the embedded trigger Sheng et al. (2022). Backdoor attacks
As previously mentioned, Carlini et al. (2023b) and Zhang et al. (2024d) demonstrate the feasibility of poisoning web-scale training data, with the latter showing that compromising just 0.1% of pre-training data is sufficient for attacks to measurably persist through post-training alignment. While Carlini et al. (2023b) specifically show this can be achieved by reverse engineering the Wikipedia snapshot process and accurately predicting when specific pages are scraped, Zhang et al. (2024d) establish that even such small poisoning rale can enable various atlacks, from denial-of-service to belief manipulation. These techniques could potentially be leveraged for Data-Centra Attacks during the pre-training stage of language models.
Harmful Pre-Training Attacks:
Harmful pest-Training Attacks: Post-Trainin Training Attacks target the instruction tuning or the preference tuning phase of model development. In Wan et al. (2023), the authors show that publicly collected datasets are prone to poisoning and that it takes as little as 100 samples for the model to exhibit a specific characteristic behavior, such as "this talentless actor," a negative polarity phrase flipped to positive polarity. Xu et al. (2023a) create a backdoor by injecting malicious instructions to activate the desired behavior without modifying the training examples or labels. In addition, the authors state, "The poisoned models cannot be easily cured by continual learning."
In Rando & Tramer (2023), the authors embed universal trigger word in the model by poisoning the RLHF training data. This trigger word acts "sudo command" and can be used during inference to produce harmful outputs. Given the multi-step process of using preferences data to train the reward model followed by the fine-tuning step, for a small model (13B paramcters), about 5% of the training data needs to be corrupted for the universal backdoor attack to survive both phases of training.
Detecting and recovering these backdoor triggers can be costly or impossible once they have infiltrated the system (Kalavasis et al., 2024). For example, the best solutions in the RLHF Trojan Competition (Rando & Tramer, 2024a) to identify injected trojans during the RLHF phase involved conducting search over the suffix space to retrieve these triggers. The first approach used the fact that there is a significant difference in the embeddings of the backdoor triggers between the compromised model and the clean model, while the second approach utilized a genetic algorithm to optimize random suffixes (Rando & Tramèr, 2024b; Andriushchenko et al., 2024; Gong et al., 2023b).
This backdoor attack serves as a critical component of red-teaming exercises, allowing teams to evaluate model vulnerabilities through intentionally planted triggers and analyze their propagation across model updates. The RLHF Trojan Competi tion (Rando & Tramèr, 2024a) highlighted several key implementation challenges: detecting backdoors in preference tuning datasets requires specialized tools like embedding comparisons and genelic algorithms, verifying model behavior demands extensive computational resources to test trigger combinations, and distinguishing malicious backdoors from benign model behaviors requires careful analysis protocols. The competition results demonstrated that even state-of-the-art detection mothods achieve limited success in identifying these vulnerabilities (Andrinshchenko et al., 2024), emphasizing the need for comprehensive testing strategies.
19

---

Published in Transactions on Machine Learning Research (05/2025)
Harmful Fine-Tuning Attacks: Open source and commercially available closed LLMs are typically fine-tuned and aligned to reduce the likelihood of generating harmful or inappropriate content. However, subsequent fine-tuning to tailor these models for specific domains or tasks can inadvertently erode this alignment, a phenomenon referred to as "Harmful Fino-Tuming Attack" (Yang et al., 2023b). This issue has garnered significant attention in the research community, as evidenced by the comprehensive survey by (Huang et al., 2024b) on harmful fine-tuning attacks and defenses.
The vulnerability to harmful fine-tuning exists in both open-souree models, where direct access to weights is available, and commercial models like GPT-4, which offer fine-tuning via API access (OpenAI, 2024a). Recent studies have shown that fine-tuning, even without explicit adversarial objectives, can undermine the original alignment (Yao et al., 2023; Yang et al., 2023b; Jain et al., 2023b; Peng et al., 2024a). Moreover, Cao et al. (2023) demonstrated the possibility of creating stealthy and persistent unalignment that resists re-alignment efforts.
The research on this topic has bifurcated into two streams: one focusing on API-gated commercial models, and the other open-source models with direct weight access. In the commercial model domain, Zhan et al. (2023) successfully circumvented RLHF safeguards in GPT-44, achieving 95% success rate with only 340 examples synthesized using less sophisticated models. For open-source models, Lermen et al. (2023) applied low-rank adaptation (LoRA) techniques to various Llama models (Touvron et al., 2023), reducing the rejection rate to below 1% on established refusal benchmarks.
Qi et al. (2023) uncovered vulnerabilities in both open-source and API-gated models, showing that safety mechanisms could be compromised with as few 10 to 100 adversarially-crafie training examples. Notably, they found that even benign finc-tuning with standard datasets could erode safety features. Bhardwaj & Poria (2023b) further demonstrated the fragility of these safety protocols, successfully manipulating responses to harmful queries at rates of 88% for ChatGPT and 91% for open-source models like Vicuna-7B (Chiang et al., 2023) and LLaMA-2-Chat (Touvron et al., 2023), using only 100 samples. While guardrail moderation is often used to mitigate harmful fine-tuning attacks, recent work by Huang et al. (2025) introduces Virus, a data optimization technique that enables attackers to construct harmful datasets capable of bypassing guardrail detection while still effectively compromising the safety alignment of fine-tuned models.
Benign fine-tuning can accidentally erase model alignment. He et al. (2024) study the examples that lead to this and state that, "Through a manual review of the selected examples, we observed that those leading a high atlack success rate upon fine-tuning often include examples presented in list, bullet-point, or mathematical formats.".
One approach to preventing this could be to erase harmful information from the model so that the model does not relearn those during fine-tuning. We discuss this in Section $ 5.2. Red teams can leverage harmful fine-tuning attacks to evaluate model robustness at two critical levels: API-gated models and open-source models with direct weight access. The assessment process requires careful consideration of: (1) minimum data requirements for successful attacks, (2) detection of unintended safety degradation from benign fine-tuning, particularly with certain data formats like lists and mathematical content (He et al., 2024), and (3) verification protocols to ensure persistence of safety features across model updates. These insights enable red teams to develop comprehensive testing strategies for both intentional and accidental compromise of model safety through funing
4.4.2 Model-Centric Attacks
Model-Centric Attacks require transparent. access to model internals to directly influence model behavior. This includes modifying weights during training (like weight tampering), but also attacks that keep weights frozen while exploiting access to internal components. For example, the adversarial prompt scarch process in Wichers et al. (2024) requires "backpropagating through the frozen safely classifier and LM to update the prompt," " making it fundamentally dependent on model access, even without weight modification.
Transparent Jailbreak Attacks: These attacks optimize adversarial prompts by backpropagating through the target model to maximize the harmful output score as measured by safety classifier. Unlike Transferable
20

---

Published in Transactions on Machine Learning Research (05/2025)
Attacks (Section 4.1.2) which optimize against proxy models, these attacks require direct access to the target model's weights and architecture. Note that methods like GCG (Zou et al., 2023) and AutoDAN (Zhu et al., 2023) (discussed in Section $ 4.1.2) could also be classified here if used directly on the main model rather than a proxy model. In Section $ 4.1.2, we categorize methods that explicitly show transferability using comprehensive experimentation. In contrast, this section classifies other transparent. optimization methods without such transferability experimentation. Nonetheless, fundamentally, both methods are quite similar as they search for adversarial prompts through a resource-intensive optimization procedure.
Representative attacks here include Wichers et al. (2024) which uses the Gumbel-Softmax trick for backprop- agation through the discrete sampling step of LMs (Jang et al., 2016; Maddison et al., 2016), while Perez et al. (2022); Deng et al. (2022) employs reinforcement learning to discover these harmful attack prompts. Alternative methods include coordinate ascent (Jones et al., 2023) and constrained decoding using Langevin dynamics (Xingang et al., 2024), both of which produce human-readable adversarial prompts, contrasting with the unnatural prompts discovered by other automated techniques. Once these attack prompts are identified during the planning phase, an adversary can attack the LLM system by employing these prompls in the execution phase via simple prompting. Since the planning phase requires transparent access to the model weights, we categorize these attacks as training attacks. In addition, the attacker might publicly distribute these exploit prompts, encouraging other malicious actors to incorporate them into their attacks.
Simple defenses like perplexity filtering can often detect and filter out adversarial suffixes generated through unconstrained optimization, as these tend to be highly unnatural sequences. Therefore, several works incorporate perplexity constraints during optimization to generate more natural-looking adversarial prompts that can bypass such filters. Furthermore, attacks in this section can extend beyond prompt optimization; for example, Qiang et al. (2023) proposes Greedy Bassed Injection (GGI) to optimize adversarial in-context examples.
In red-teaming exercises, Transparent Prompt Search Attacks can systematically probe model vulnerabilities by directly optimizing against safety objectives. However, its computational intensity and access requirements typically limit its use to internal red teams or dedicated security researchers. Key consideration ions include choosing appropriate safety classifiers that align with deployment risks, balancing optimization constraints to generate rcalistic attack vectors, and carcfully documenting discovered vulncrabilities for defense development.
Adapters and Model Tampering Attack: In addition to creating backdoors by poisoning training data Feng & Tramer (2024) lamper with model's training weights to compromise privacy of fine-tuning data while Dong et al. (2023) train trojan adapters using low-rank adaptation.
4.5 Compound Systems Attack
AT Systems are rapidly compounding with multiple components (Zaharia el al., 2024). This could be visualized as multiple replicas of Figure 2 interacting with each other. LLMs can also be combined with tools or other LLMs to form complex agents (Wang et al., 2023d; Zaharia et al., 2024; LLM Agents, 2023; Hamilton, 2023; Mei et al., 2024a). These compound systems introduce novel vulnerabilities that require careful consideration for safety (Tang et al., 2024; Fang et al., 2024b). Yang et al. (2024) investigate backdoor attacks, while Mo et al. (2024a) provides a valuable framework to conceptualize and map adversarial attacks against Language- Based Agents. Multiple methodologies exist for the construction of multi-agent systems. A notable approach is collaboration via debate, which has been shown to enhance the factual accuracy and reasoning capabilitics of multi-agent systems (Du et al., 2023). However, Amayuclas et al. (2024) show that a persuasive adversarial agent can compromise this system. More recently, Cohen et al. (2024) introduced the first worm designed for GenAl ecosystems. Here, the attacker inserts "adversarial self-replicating prompts" into the input. These prompts can spread maliciously by inducing generative LLM to replicate these harmful prompts in their output. Furthermore, guardrail models, often used to filter out harmful responses in compound LLM systems, are also susceptible to attacks (Mangaokar et al., 2024).
The testing of compound LLM systems demands a fundamentally different sccurity mindset than the evaluation of standalone models. Beyond individual vulnerabilities, security teams must probe for emergent attack vectors arising from component interactions whether through manipulated agent cooperation, compromised
21

---

Published in Transactions on Machine Learning Research (05/2025)
communication channels, cascade failures that spread across the system. The discovery of self-replicating prompis and persuasive adversarial agents illustrates how novel threats can emerge at the system level that. would be impossible to detect by testing components in isolation.
5 Defenses
In this section, we provide high-level overview of current defense methodologies by highlighting key papers in the field. Defense stralegies against LLM attacks can be broadly categorized into three approaches: extrinsic, intrinsic, and holistic defenses. Extrinsic defenses operate without modifying the model, typically through input/output filtering or prompt engineering. Intrinsic defenses involve changes to the model itself through techniques such as alignment or adversarial training. Holistic defenses combine multiple approaches leverage system-level protections. This categorization allows us to systematically analyze defense methods across different attack vectors, while acknowledging that many defense strategies can protect against multiple types of atlack simultaneously.
Numerous other papers and resources offer a comprehensive overview of defense techniques (Mozes et al., 2023; Dong et al., 2024b; Xu et al., 2024b; Dong et al., 2024a; LLM Security, 2023). Table 5.3 provides a high-level overview of various defense strategies.
5.1 Extrinsic Defense
Prompt Based Defense: For prompt-based attacks, defenses can target just the prompt or prompt-model_ouee outptt The first line of defense is to verify them against standard content moderation APIs and guardrails (OpenAI moderation endpoint (OpenAI, 2024b; Markov et al., 2023), Azure Content Safety API, Perspective API (Lees et al., 2022), Llama-Guard (Inan el al., 2023), RigorLLM (Yuan al., 2024), Nvidia Nemo (Rebedea et al., 2023), Guardrails AI (Guardrails AI, 2024)). However, Glukhov et al. (2023) point out, the detection of undesirable content using another model has theoretical limitations due to the undecidable nature of censorship.
On the prompting front, there are several techniques to mitigate jailbroaks such as SmoothLLM (Robey et al., 2023), adding an H& directive or self-reminders (Xie et al., 2023), Intention Analysis Prompting (Zhang et al., 2024e), Robust Prompt Optimization (Zhou et al., 2024a), Prompt Adversarial Tuning (Mo et al., 2024b), Backtranslation (Wang et al., 2024c), Moving Target Defense (Chen et al., 2023a), Semantic Smoothing (Ji et al., 2024), Self-Defense (Helbling et al., 2023) and Paraphrasing (Yung et al., 2024). Since prompt injection attacks are similar to SQL injections, parameterizing prompt components, such as separating input from instructions and adding quotes and additional formatting, could also serve as viable defense techniques (Hines et al., 2024; Chen et al., 2024b; Adversarial Prompting, 2023).
Perplexity filtering is another simple defense against adversarial suffixes obtained by optimization, since these tend be non-natural language-like phrases (Alon & Kamfonas, 2023; Qi et al., 2020; Hu al., 2023). Model pruning (Hasan et al., 2024) and safe decoding (Xu et al., 2024a) have also been shown to improve resistance to jailbreak prompts, which can be applied post hoc. Jain et al. (2023a) evaluate several of the baseline defenses and discuss their feasibility and effectiveness.
Prompt-based defenses represent the blue team's first line of protection against Jailbreak Attacks (Section $ 4.1.1) and Direct Attacks (Section 4.1). These defenses span the full security capabilities matrix: prevention through input validation and filtering, detection via content moderation APIs, mitigation using fallback responses, and recovery through logging and incident response. Importantly, these extrinsic defenses require minimal access to the underlying model, making them particularly suitable for applications built on black-box LLM APIs. When red-teaming LLM applications, it is crucial to evaluate them with all guardrails activated, as this reflects real-world deployment conditions. Red teams can systematically probe defense effectiveness by testing different prompt allack strategies against multiple protective layers, while blue teams iteratively strengthen these defenses based on discovered vulnerabilities. This creates a continuous improvement cycle- for instance, while perplexity filtering may catch optimized adversarial suffixes, red teams might discover more sophisticated attacks using human-readable text that bypass this defense, prompting blue teams implement additional protective measures like semantic analysis. Most current red-teaming research focuses
22

---

Published in Transactions on Machine Learning Research (05/2025)
this iterative strengthening of defense mechanisms, particularly for these extrinsic defenses due to their broad applicability and ease of implementation in production systems.
Certifications: When faced with malicious prompts in real-world scenarios, it is difficult to assess the level of risk or harm posed by an LLM. Kumar et al. (2023) introduce a framework called erase-and-check to address this issue by offering defenses with certifications against three attack modes: (i) adversarial suffix, (ii) adversarial inscrtion, and (iii) adversarial infusion. These defense mechanisms work by crasing a portion of the original prompt to create several subsequences and checking each of them with a safety filter. If any of the subsequences are classified as harmful by the safety filter, the main prompt is marked as harmful.
To mitigate the PoisonedRAG attack, Zou et al. (2024) (see Section $ 4.2) introduces an "isolate-then aggregate" strategy, which offers a certifiable robustness guarantee. The principal innovation in their defense mechanism lies in isolating the retrieved passages prior to invoking an LLM for responses, followed by a secure aggregation of these isolated responses. This methodology ensures that the influence of a limited number of malicious passages is confined to correspondingly limited subset of isolated responses. In addition, Kang et al. (2024a) propose C-RAG, a framework that provides conformal risk analysis for RAG models and certifics an upper confidence bound of generation risks, showing that RAG achieves lower conformal generation risk than single LLMs when the quality of retrieval and transformer is non-trivial.
Additionally, prior research has investigated certified robustness techniques for classification tasks (Huang et al., 2024f; Zhang et al., 2023b; Ye ct al., 2020; Zhao et al., 2022). However, these techniques do not directly apply to the LLM generation setting and could be a valuable area for future investigation. Vulnerability scanners and databases are another emerging paradigm that can also help with the continuous monitoring and reporting of LLM harms (Derczynski et al., 2024; AI Vulnerability Database, 2023; Giskard, 2024).
Certification methods are particularly valuable for defending against Infusion Attacks (Section & 4.2) and automated attack methods (Section $ 4.1.2). Unlike prompt-based defenses, certifications require deeper integration with the model architecture but can still be implemented on top of black-box APIs through wrapper frameworks. For instance, the maase-and-chee framework provides certified defense against adversarial suffixes without requiring model access, while solate-then-aggreaate strategies protect RAG systems from poisoned retriovals. When red-tcaming certified systems, blue teams can leverage these formal guarantees to establish clear safety boundaries, while red teams focus on finding edge cases that might violate the certification assumptions. Most current research in this area focuses on expanding certification techniques to cover more complex attack sconarios while maintaining practical computational overhead.
5.2 Intrinsic Defense
Alignment through preference tuning represents a powerful intrinsic defense mechanism for LLMs, aiming to build safety and security directly into model behavior rather than relying on external filters (Ziegler et al., 2019; Ahmadian et al., 2024; Chen et al., 2024a; Gulcehre et al., 2023; Munos et al., 2023; Liu et al., 2023c; Wang et al., 2023b; Ethayarajh et al., 2024; Zhao al., 2023). The foundational work by Ouyang et al. (2022b) demonstrated that reinforcement learning from human feedback (RLHF) could effectively prevent models from generating harmful content, protecting against wide range of potential misuse. This approach was further developed by Askell et al. (2021), who showed that constitutional Al principles could create models that maintain security while remaining helpful, honest, and harmless (H3).
Alignment:
Several approaches have emerged to scale and improve alignment techniques. Brown-Cohen et al. (2024) propose dobate as a scalable method for aligning AI systems. Leike ct al. (2018) introduce recursive reward modeling to handle increasingly complex tasks. Recent work by Saunders et al. (2022) demonstrates how to use self-critique to improve model behavior.
As a defensive technique, alignment offers several key advantages. Huang et al. (2023a) show that generation- aware alignment can build robust defenses against generation-exploitaaaa attacks described in Section & 4.1.4. As described in Section $ 4.1.1, competing objectives can lead to higher attack success rates. To counteract this, Zhang et al. (2023d) demonstrate how goal prioritization during training and inference can create strong safety boundaries when different objectives conflict. These methods provide defense- -nnnsst----epthh by embedding security constraints directly into model weights rather than relying on post-hoc filtering.
23

---

Published in Transactions on Machine Learning Research (05/2025)
However, current alignment techniques face important security limitations that need to be addressed. Models can exhibit overly conservative behaviors (Röttger et. al., 2023; Bianchi et al., 2023), refusing benign requests (e.g., "How do I make someone explode with laughter?") due to excessive dependency on lexical cues. To strengthen these defenses, Wallace et al. (2024) propose incorporating instruction hicrarchics that provide more robust protection against malicious prompls attempting to override safety constraints. Additionally, significant research has focused on preventing hallucinations which can pose security risks (Tian et al., 2023; Sennrich et al., 2023; Dhuliawala et al., 2023; Li et al., 2023d; Zhang et al., 2024c).
Recent theoretical work by Peng et al. (2024b) has advanced our understanding of alignment's security properties, discovering "safety basin" phenomenon where model behavior remains safe under small parameter perturbations but degrades sharply beyond certain bounds. Red teams have found that while alignment provides strong baseline protections, current methods can exhibit brittle safety properties that may be bypassed through latent space manipulation (Arditi et al., 2024). This aligns with our discussion in Section $ 7 about aligmment limitations and deceptive behavior (Hubinger et al., 2024; Greenblatt et al., 2024). Unlike extrinsic defenses, aligmment methods require full model access, making them particularly suitable for implementation by model providers during the training process. Future work should focus on developing more robust alignment techniques that maintain strong security guaranteos while preserving model utility.
Adversarial Training: Empirical risk minimization used in supervised fine-tuning or policy gradient methods used in aligument do not lead to models that are robust to adversarial attacks. Carlini et al. (2023c) conjecture that, "improved NLP attacks may be able to trigger similar adversarial behavior on alignment-trained text-only models." Adversarial training is often used to improve robustness against such attacks. Vanilla adversarial training involves solving a min-max optimization in which the objective of the inner maxinization is to perturb the data point to maximize the training loss, while the outer minimization aims to reduce the loss resulting from the inner attack (Madry et al., 2017). Automatically generating these perturbations in the latent space is challenging for discrete text space; thus, adversarial techniques in NLP generally involve human generated adversarial examples or creating automatic adversarial perturbations by adjusting at the word, sentence, or syntactic level to mislead the model, but remain imperceptible to humans (Wang et al., 2021).
Building on this, Ziegler et al. (2022) introduce the notion of "high-stakes reliability" and utilize an adversarial training approach to improve defense against attacks in safe language generation task. Casper et al. (2024c) presents latent adversarial training (LAT) to protect against new vulnerabilities that differ from those at training time, without creating the adversarial prompls that cause them. Several approaches have been developed to specifically defend against harmful fine-tuning and maintain model safety. Rosati et al. (2024b) introduced "Immunization conditions" that establish constraints during finc-tuning to preserve safety properties, while Wang et al. (2024a) developed "Backdoor Enhanced Safety Alignment" to proactively defend against. potential jailbreak attacks. Rosati et al. (2024a) proposed " "Representation Noising" to eliminate harmful information from model weights, preventing the relearning of unsafe behaviors during fine-tuning.
Recent work has further expanded the toolkit for defending against harmful fine-tuning (Section $ 4.4.1). Hsu et al. (2024) proposes Safe LoRA, which projects LoRA weights onto a safety-aligned subspace derived from aligned and unaligned model checkpoints, demonstrating effective defense against harmful fine-tuning without requiring additional training data. Vaccine (Huang et al., 2024e) uses perturbation-awar alignment. to vaccinate models during training, while Lisa (Huang et al., 2024d) maintains alignment through two-state optimization balaneing safety and performance. For post-hoc remediation, Antidote (Huang et al., 2024a) employs targeted pruning to remove harmful weights, and T-Vaccine (Liu et al., 2024a) selectively perturbs critical layers. Booster (Huang et al., 2024c) attenuates harmful perturbations through regularization, achieving state-offthe-art results. Together, these techniques provide ML practitioners with preventive training and post-hoe fixes for protecting ing model alignment while managing computational costs.
Furthermore, Zeng et al. (2024c) propose Backdoor Embedding Entrapment and Adversarial Removal (BEEAR) which mitigates backdoor attacks discussed in Section $ 4.4.11 Their main finding is that backdoor triggers produce a "uniform drift" in the model's 's embedding space, irrespective of the specific type or purpose
24

---

Published in Transactions on Machine Learning Research (05/2025)
of the trigger. They suggest a bi-level optimization algorithm, with the inner level identifying this universal drift and the outer level adjusting the model parameters for safe behavior.
Privacy: The canonical approach to privacy in ML is to use differentially private methods (Dwork & Roth, 2014; Abadi et al., 2016; Song et al., 2013; Bassily et al., 2014; Carlini et al., 2018; Ramaswany et al., 2020; Anil et al., 2021; Yu et al., 2021a; Li et al., 2021c; Yu et al., 2021b). Ye et al. (2022) introduce an efficient differentially private method to protect against both membership inference and model inversion attacks with minimal parameter tuning. Ozdayi et al. (2023) investigate the application of prompt tuning to modulate the extraction rates of memorized content in LLM, offering a novel strategy to mitigate privacy risks (Smith et al., 2023). Adding defensive approaches, Gong el. al. (2023a) proposed GAN-based (Generative Adversarial Networks (Goodfellow et al., 2014)) method to counteract model inversion attacks images. Data deduplication is another common strategy to improve privacy and memorization risks in language models (Lee et al., 2021; Carlini et al., 2022; Kandpal et al., 2022), however, this also opens up a privacy side channel as discussed previously in Section $ 4.1.4.
In a red-teaming setup, blue teams can leverage several of these defenses: Adversarial training methods (LAT (Casper et al., 2024c), BEEAR. (Zeng et al., 2024c)) and privacy mechanisms (DP (Ye et al., 2022), deduplication (Lee et al., 2021; Carlini et al., 2022)) should be proactively applied to harden models against. Direct (S 4.1), Backdoor ($ 4.4.1), Inversion ($ 4.1.3), and Side-Channel ($ 4.1.4) Attacks.
Defenses like Vaccine (Huang et al., 2024e), Lisa (Huang et al., 2024d), Antidote (Huang et al., 2024a), T-Vaccine (Liu et al., 2024a), and Booster (Huang et al., 2024c) can protect aligned models from harmful fino-tuning attacks (6 4.4.1). Based on red team findings, blue teams should iteratively tunc defense parameters to optimize the robust robusstess--ullllyy-ppie puiiity-privacy trade-off and systematically re-test mitigations.
5.3 Holistic Defense
Multi-Layered Defense: In practical applications, it is crucial to combine several defense strategies to improve overall effectiveness in reducing potential thrcats. This method resembles the Swiss cheese model (SCM) frequently applied in fields like cybersecurity, aviation, and chemical plant safely (Reason, 1990), where multiple layers of defense collectively provide enhanced protection against adversarial threats. For example, several guardrails can be stacked on top of each other to filter out harmful prompts and output from an LLM. A combination of intrinsic and extrinsic defense methods can also be employed simultancously to enhance security. This method is expected to be effective because the mistakes of different guardrails and defense techniques are likely to be uncorrelated. As far as we are aware, only a limited number of prior studies have suggested a multi-layered defense approach for LLMs (Rai et al., 2024; Pienaar & Anver, 2023), making this a potentially valuable area for future research.
The notion of employing multiple layers of defense exlends to the use of multiple agents. For example, Zeng et al. (2024d) introduces "AutoDefense", a system that utilizes several LLMs to mitigate jailbreak attacks. The overall defense task is divided into smaller subtasks, cach delegated to different LLM agents. This division allows LLM agents to focus on specific elements of the defense st strtegyy, such as assessing the intent of the response or making the final decision, leading to better performance. A different study by Ghosh et al. (2024) supports the use of an ensemble of experts, whose weights are adapted in real-time through an online algorithm. In the same vein, Lu et al. (2024b) presents the idea of "mixxure-of--eefeddee
Multi-Agent Defense:
(MoD), where each expert is focused on tackling a specific type of jailbreak
prompt.
Other Design Choices: Defense strategies can go beyond reactive methods like guardrails and tuning- based approaches such as aligmment. Occasionally, it is necessary to reconsider model architecture decisions, API parameters, or training algorithrns. For example, to mitigale altacks originating from the "Sofumax Bottleneck" as discussed in Section $ 4.1.4, Finlayson et al. (2024) outlines three strategies. The first strategy, applicable when the model reveals log probabílitics, is to restrict access to these log probabilities. However, as mentioned earlier, Morris et al. (2023a) demonstrate that complete probabilities can still be inferred by leveraging API access to Logit bias through binary search algorithm requiring O(v log e) APT calls. Here
25

---

Published in Transactions on Machine Learning Research (05/2025)
is the vocabulary size. However, the inefficiency of the algorithm can serve as deterrent, noted in Finlayson et al. (2024), "Regardless of the theoretical result, providers can rely on the extreme inefficiency of the algorithm to protect the LLM. This appears to be the approach OpenAl took after learning about this vulnerability from Carlini et al. (2024), by always returning the top-k unbiased logprobs." Finlayson et al. (2024) further improve the algorithm to O(d log <) API calls, which, their opinion, undermines the case for algorithmic inefficiency-baseed defenses. Here d the hidden dimension size. The second strategy involves completely eliminating the Logit bias parameter. This method is more robust since there are no existing techniques to retrieve full probabilities without Logit bias. Most LLM providers and latest OpenAl models do not expose the Logit bias parameter in their APIs. The third proposed strategy involves moving towards model architectures that are inherently free from the Softmax bottleneck.
To mitigate attacks on MoE models Hayes et al. (2024) proposes randomizing examples in the batch, using a large buffer, and introducing stochasticity in expert assignments for a token.
The defense strategies described above provide a robust toolkit for blue teams to counter the diverse range of attacks outlined in Section $ 4. In red-teaming exercise, the red team probes various entry points using techniques like Jailbreak Attacks ($ 4.1.1), Direct Attacks ($ 4.1), and Side-Channe Attacks ($ 4.1.4), the blue team can employ combination of guardrails, diverse defender agents, and strategic architectural decisions to build resilience. For instance, stacking multiple content filters helps catch harmful prompts that slip through a single filter. Delegation to specialized defender agents enables targeted protection against specific jailbreak strategies. Careful choices like restricting access to log probabilities or using bottleneck- architectures proactively close exploit avenues. By tactically combining these holistic approaches, blue teams can comprehensively stress-test and fortify the LLM system against the red team's 's creative attacks, ultimately delivering more secure and robust model.
6 Towards Effective Red- Teaming
Having explored various attack vectors and defense strategics, it is crucial to understand how these clements come her in effective red-teaming practices. While individual allacks target specific vulnerabilities, red- teamingg represents a systematic approach to security assessment that integrates diverse testing methodologies throughout the model development lifecycle. This broader perspective on socurity evaluation has led researchers and practitioners to develop structured frameworks and best practices to conduct comprehensive red-teaming exercises. In this section we examine the key components that contribute to effective red-teaming of LLM systems.
Feffer et al. (2024) identify several challenges plaguing current red-teaming efforts, ranging from lack of consensus on what should be tested, who should be testing, to unclcar follow-ups to red-teaming exercises. They offer a question bank that acts as a framework for future red-teaming exercises. They divide these questions into pre-activity, include the identification of the threat model, the specific vulnerability under consideration, the level of access granted to participants, and the success critcria for the red-teaming exercise. Our threat model and attack taxonomy provide answers to some of these questions.
Red-Teaming Question Bank:
activity, and post-activity. Some of these questions
Multi-Round Automatic Red-Teaming: In line with conventional rod-teaming practices, several recent works have suggested iterative frameworks that are based on multiple rounds of attack and defense interactions between red-tcam language models (RLMs) and blue-team language models (BLMs) (Ge et al., 2023; Mchrabi et al., 2023b; Ma al., 2023; Li et al., 2023d; Xiao al., 2024; Jiang et al., 2024; Zhou et al., 2024b). Li et al. (2024b) demonstrate that despite these automated approaches, human red teamers still significantly outperform automated methods in multiturn settings, suggesting that current automated frameworks may not fully capture the sophistication of human attack strategios.
Uncovering Diverse Attacks: Only a few previous studies have attempted to visually represent the semantic regions of successful attacks (Perez et al., 2022). These are basod mainly on clustering-based techniques. Kour et al. (2023) enhance the quality of the semantic regions detected by developing a novel honogeneixx-presevvin clustering technique. They found that human-generated attacks exhibit diversity,
26

---

Published in Transactions on Machine Learning Research (05/2025)
Study
Category
Short Description
Free
Extrinsic
(OpenAT, 2024bb
Guardrail
OpenAT Moderations Endpoint
K
(Lees et al., 2022)
Guardrail
Perspective API's Toxicity API
X
(Inan al., 2023)
Guardrail
Llama Guard
(Guardrails AI, 2024)
Guardrail
Guardrails AI Validators
(Rebedea al., 2023)
Guardrail
NVIDIA Nemo Guardrail
(Yuan et al., 2024)
Guardrail
RigorLLM (Safe Suffix + Prompt Aug-
mentation + Aggregation)
(Kim et al., 2023)
Guardrail
Adversarial Prompt Shield Classifier
(Han et al., 2024)
Guardrail
WildGuard
(Robey et al., 2023)
Prompting
SmoothLLM (Prompt Augmentation
Aggregation)
(Xie al.; 2023)
Prompting
Sell-Reminder
(Zhang et al., 2024e)
Prompting
Intention Analysis Prompting
(Wang et al., 2024c)
Prompting
Backtranslation
(Zhou et al., 2024a)
Prompting
Safe Sullix
(Mo al., 2024b)
Prompting
Safe Prefix
(Chen et al., 2023a)
Prompting
Prompt Augmentation Auxiliary
model
(Ji et al., 202-1)
Prompting
Prompt Augmentation + Aggregation
(Yung et al., 2024)
Prompting
Prompt Paraphrasing
(Alon & Kamfonas, 2023)
Prompting
Perplexity Based Defense
(Liu et al., 2024b)
Prompting
Rewrites imput prompt to safe prompt using sentinel model
(Xiong et al., 2024)
Prompting
Safe Suflix (Requires access to (og-probabilities
(Liu et al., 2024f)
Prompting
Information Bottleneck Protector
(Suo, 2024)
Prompting/
Introduces Signed-Prompp for autho-
Tuning
rizing sensitive instructions from
proved users
(Xu et al., 2024a)
Decoding
Safety Aware Decoding
(Hasan et al., 2024)
Model Pruning
Uses WANDA Pruning (Sun et al., 2023)
(Yi et al., 2024)
Model Merging
Subspace-oriented model fusion
(Arora et al., 2024)
Model Merging
Model Merging prevent backdoor tacks
(Stickland et al., 2024)
Activation Editing
KL-tten-steer to decrease side-effects of steering vectors
(Huang et al., 2023a)
Alignment
Generation Aware Alignment
x xx
(Zhao et al., 2024b)
Alignment
Layer-specific editing
(Qi et al., 2024)
Aligument
Regularized Eine-tuning objective for deep safety aligument
(Zhang al., 2023d)
Alignment
Goal Prioritization during training and inference stage
(Ghosh et al., 2021)
Alignment
Instruction tuning on AEGIS safety dataset
(Wallace et al., 2024)
Fine- Fine-Tuning Fine-Tuning
Training with Instruction Hierarchy
(Rosati al., 2024bb
Immunization Conditions prevent against harmful fine-tuning
(Wang et al., 2024a)
Fine-Tumie Fine-Tuning
Backdoor Enhanced Safety Alignment to prevent against harmful fine-tuning Representation Noising against harmful fine-tuning
(Rosati et al., 2024a)
Fine-Tuning
prevent
X
(Yu et al., 2021a)
Fine- Tne-Tunnng
Differentially Private fine-tuning
(Xiao et al., 2023)
Fine-Tuning
Privacy Protection Language Models
(Casper et al., 2024c)
Fine-Tiie Tine-Tuning
Latent Adversarial Training
27

---

Published in Transactions on Machine Learning Research (05/2025)
Study Category
Short Description
Free Extrinsic
(Liu et al., 2023b)
Fine-Tuning
Denoised Exopuct-of-xxperss for pro- tecting against various kinds of back- door triggers
X
(Wang al., 2024b)
Fine-Tuning
Detoxifying by Knowledge Editing of Toxic Layers
x
(Huang al., 2024e)
Fine- Tunnn
Vaccine uses perturbation-awark align- ment to vaccinate models during train- ing
x
(Liu et al., 2024a)
Fine- Tuning
Vacecee selectively perturbs critical layers for post-hoc protection Safety-critical parameter gradients anal- ysis
x
(Xie al., 2024b)
Inspection
x
(Kumar et al., 2023)
Certification Certification
Erase-and-check framework
(Zou et al., 2024)
Isolate-then-Aggreen
to protect
against AAioonedRAGAttaan Bias Certification of LLMs
(Chandhary et al., 2024)
Certification Certification
(Kang al., 2024a)
Certifies an upper confidence bound of generation risks in RAG setting Garak LLM Vulnerability Scanner Evaluate Performance, Bias issues in AI applications
(Derezynski et al., 2024)
Model Auditing
(Giskard, 2024)
Model Auditing
Table 4: (Contimued from previous page) Summary of several defense strategies that can serve as a general guide for practitioners. Prompt Augmentation entails paraphrasing to generate multiple versions of prompts, whereas Aggregation merges outcomes from several LLM queries. Fine-Tuning methods necessitate transparent access to model weights and include some degree of fine-tuning of the target model. Prompting methods might involve fine-tuning but typically on an auxiliary model and do not need access to the model weights.
which includes several aspects of harmfulness, whereas generative monel-gen attacks display a high degree of clustering. Hong et al. (2024) overcome this limitation of automatic red-tcaming by proposing a "curiosity-driven exploration" that generates more diverse test cases, while Sinha et al. (2023) propose an adversarial training framework that can generate useful adversarial examples at scale using small number of human adversarial examples. Additionally, Samvelyan et al. (2024) introduce "Rainbow Teaming" for generating diverse adversarial prompts by framing prompt generation as quality-diversity problem. Recent work has made significant progress in combining automated and human approaches Beutel et al. (2024) demonstrate that cule-based rewards combined with multi-step reinforcement learning can generate attacks with diversity comparable human red tcamers, while Ahmad et al. provide structured methodology for balancing automated tools with domain expert evaluations to achieve comprehensive coverage of potential vulnerabilities.
Automated techniques and tools have limited diversity that hinder their ability to identify catastrophic errors with the same accuracy and precision as humans. Therefore, human annotators remain crucial part of red-teaming exercises (Ropers et al., 2024).
Taxonomy-Free vs. Taxonomy-Guided Red-Teaming: Red-teaming approaches can be taxonomy- free haxonomy-guidedd In taxonomy-oo free red-teaming, practitioners first carry out attacks (manually automatically) to characterize the risk surface (Ganguli et al., 2022b). Taxonomy-guided red-teaming begins with constructing risk taxonomy, then directs experts to probe specific risks (Weidinger et al., 2021). A hybrid approach combining both methods can help uncover risks not covered in existing taxonomies. As discussed Section $ 2.2, while LLM providers' risk taxonomies offer initial frameworks, domain-specific expansions are often necessary. For example, financial or legal applications may prioritize hallucination risks over safety failures, requiring adapted taxonomies. Furthermore, as LLM systems become more complex with diverse artifacts, risk taxonomies must. evolve to address emerging vulnerabilities.
28

---

Published in Transactions on Machine Learning Research (05/2025)
Effect of Scaling on Red-Teaming: Ganguli et al. (2022b) investigated the scaling behavior of red- teaming across different model sizes and model types and released dataset of 38,961 attacks. They find that larger aligned models (trained with Reinforcement Learning from Human Feedback) are considerably harder to attack compared to plain LM, LMs prompted with H2 directive, and best-of-n sampling from an LM. Other models exhibit, a flat trend with scale. In contrast to previous results, they found that H3 prompting is not always an effective defense mechanism and that best-of-n sampling is relatively robust to attacks. Recent work (Ren et al., 2024) highlights that many safety benchmarks correlate strongly with model capabilities. To avoid confusing capability improvements with safety gains, practitioners should empirically validate whether safety metrics measure distinct properties versus proxying capabilities, report capability correlations for new evaluations, and develop benchmarks that isolate safety-specific attributes.
Remediating Attack Recurrence: Attacks may be patched through the use of filters (extrinsic defense) or fine-tuning (intrinsic defense). In some cases, the patches put in place might be too narrow or regress again with future version of the model (Breitenbach & Wood, 2024).
Recent work has proposed several benchmarks in response to the need for a standardized evaluation process for red-teaming, which can also help avoid unintended regressions (Li et al., 2024a; Zhang et al., 2023c; Bhardwaj & Poria, 2023a; Mazeika et al., 2024; Microsoft, 2023; Chen et al., 2023c; Wu et al., 2024a; Wang et al., 2023a; Pattnaik et al., 2024; HaizeLabs, 2024; Chao et al., 2024; Tedeschi et al., 2024; Zou et al., 2023; Hung et al., 2023; Yi et al., 2023; Wang et al., 2024b). Furthermore, numerous jailbreaks detected by existing evaluation techniques could be false positives, where hallucinated responses are incorrectly considered actual safety violations, highlighting the need for stricter and more accurate benchmarking criteria (Mei al., 2024b; Xie et al., 2024a).
7 Discussion and Implications
The rapid proliferation of LLM-based applications has presented a unique set of new challenges for red-teaming (Zhu et al., 2024). For example, Chen et al. (2023b) discuss the issue of prompt drift, where prompts that were once successful in attacking a model may not work in the future. Furthermore, certain behaviors could be dual intent (e.g., generating toxic outputs to train a toxicity classifier) (Mazeika et al., 2024; Stapleton et al., 2023). Similarly, classifiers used to identify harmful outputs in automated red-teaming could have their own biases and blind spots (Perez et al., 2022). The use of LLM as a judge is also prone to a variety of adversarial attacks (Raina et al., 2024). Filtering- based defenses, such as memorization filters, aim to prevent LLMs from generating copyrighted content but might inadvertently create new vulnerabilities in the form of side channels. Multiple adversaries may also attempt to poison the same dataset or target multiple entry points simultaneously, which is a valuable direction for future investigations (Graf et al., 2024).
Alignment, which is considered a standard technique for AI safety, also suffers from several limitations (Wolf et al., 2023). For example, Hubinger et al. (2024); Greenblatt et al. (2024); Vaugrante et al. (2025) show that models can be trained to act deceptively while appearing benign under standard safety training. Alignment techniques can often lead to exaggerated safety behaviors (Röttger et al., 2023; Bianchi et al., 2023). Evaluating the trade-offs between evasiveness and helpfulness could be potential direction for future investigations.
Human preferences are incorporated into an LLM during the model alignment phase by querying a reward model (RM). A good reward model provides high numerical scores for H3 generations and low numerical scores for toxic, discriminatory, or harmful content, thus reinforcing good behavior a model and penalizing bad behavior. Harandizadeh et al. (2024) study if the reward model penalizes certain categories of risk more rigorously against others. They find that compared to other harm categories (e.g., "Malicious Use", "Discrimination /Hateffu /Hateful content"), RMs perceive "Information Hazards" (exposing personally identifiable information) as less harmful (Harandizadeh et al., 2024). This observation presents a challenge for equitable model alignment across various harm categories.
As LLMs become more autonomous and gain broader capabilities, more cybersccurity threats are expected to penetrate inteerttee applications. In addition, adversaries could utilize more covert injections divided into several attack phases, in which a preliminary prompt injection directs the model to retrieve a larger
29

---

Published in Transactions on Machine Learning Research (05/2025)
payload (Greshake et al., 2023). Model modalities also expected to increase (e.g., GPT-4), which could also open new doors for injections (e.g., prompts hidden in images).
7.1 Takeaways
Based on analysis of the challenges, limitations, and emerging threats in LLM security, we can distill several key recommendations for practitioners implementing red-tcaming in real-world applications. These takeaways synthesize insights from both current best practices and anticipated future needs in the rapidly evolving landscape of LLM security.
Domain-Specific Risk Taxonomy: While publicly available risk taxonomies provide solid founda- tions, practitioners should customize them based on domain-specific concerns (Section $ 2.2). For example, hallucination risks are often under-emphasized in standard taxonomies but may be critical for domains like retrieval-angmented generation, legal analysis, and financial services where factual accuracy is paramount. Applications should adapt taxonomics to prioritize risks relevant to their use case while maintaining standard safety considerations.
Current red-teaming efforts often focus narrowly on prompt-based jailbreaks ($ 4.1.1) and direct allacks (8 4.1). However, our laxonomy reveals a much broader attack surface, including model inversion (6 4.1.3), side channels ($ 4.1.4), harmful fine-tuning attacks (S 4.4.1), and system-level vulnerabilitics (S 4.5). Practitioners should employ in-depth defense strategies (Mughal, 2018), employing holistic defense strategies as described earlier ($ 5.3). They should proactively consider supply chain attacks on training data and model weights ($ 4.1.3) and monitor side channels arising from architectural decisions and deployment choices ($ 4.1.4). Just as cybersecurity involves securing all networking layers, LLM security requires investigating all attack vectors across the model lifecycle. Developing tools to facilitate such comprehensive red-teaming is crucial for robustly securing LLM systems against the full spectrum of threats outlined in our taxonomy.
Broadening Beyond Prompt-Based Attacks:
Diversity in Red-Teaming: Red-teaming exercises should combine both manual and automated approaches, as this enables greater semantic diversity in discovering vulnerabilities (Section & 6). This is particularly important given the increasing sophistication of attack prompts over time (Section & 4.1.1). Building and maintaining community of trusted red-tcamers can accelerate the identification of novel attack vectors (OpenAI, 2023).
Importance of Inclusive Red-Teaming: Red-tcaming excrcises should incorporate diverse perspectives and structured evaluation protocols. This includes matching evaluator demographics to harm categories, implementing multi-layer annotation with arbitration for nuanced assessment, and using parameterized testing instructions for comprehensive risk coverage (Weidinger et al., 2024). These sociotechnical considerations help identify potential harms that might be overlooked by purely technical evaluations.
Documenting Overrefusals in Red-Teaming For LLMs, models that reject all potentially sensitive queries create an illusion of security while eliminating practical utility. Röttger et al. (2023) show how aligned models frequently refuse harmless requests containing adfety-adjacent terminology echoing traditional security-usability tensions where overly strict controls drive users toward workarounds, ultimately compromising safety (Garfinkel & Lipford, 2014). Complete red- teaming protocols must therefore document both successful attacks and false rejections, mcasuring the essential balance between protection and functionality that determines real-world effectiveness.
Systematic Vulncrability Tracking and Defense Evolution: Drawing parallels from cyberse- curity practices around CVE tracking and vulnerability databases (Mell et al., 2007), practil ioners should maintain comprehensive records of attack patterns, their variations, and mitigation strategies across model architectures. Recent initiatives like the AI Vulnerability Database (AI Vulnerability Database, 2023) and ATLAS Matrix (ATLAS Matrix, 2023) provide frameworks for this systematic approach. The demonstrated success of transfer attacks from open-source to commercial models
30

---

Published in Transactions on Machine Learning Research (05/2025)
(Section $ 4.1.2) through methods like GCG and AutoDAN underscores the importance of docu- menting attack transferability. This documentation, combined with certification methods (Section $ 5.1), creates crucial feedback loops between vulnerability discovery and defense improvements, helping prevent regression of patchod vulnerabílities (Section & 6) while cnabling continuous security enhancements across the LLM ecosystem.
Supply Chain Security for LLMs:
Drawing from software supply chain security principles in NIST's Secure Software Development Framework (Souppaya et al., 2022), practitioners must secure the entire LLM development pipeline. Our analysis of infusion attacks (Section $ 4.2) demonstrates how compromised data sources and external tools can inject harmful behaviors. Similar to the access management and sandboxing techniques used in cybersecurity, isolating tool execution and mediating interactions can roduce security and privacy risks from Infusion Attacks (Wu et al., 2024b) (Section $ 4.2). The rise of model weight. lampering and backdoor attacks (Section S 4.4.1) further emphasizes the need for rigorous integrity verification of model artifacts throughout the development lifecycle.
System-Level Security for Compound LLMs:
Drawing from cybersecurity defense-in-den in-tepth prin- ciples JointTaskooree 2017; Mughal, 2018), practitioners must implement multi-layered protections for increasingly complex LLM applications ($ 4.5). Recent analyses of multi-stage attacks (Cohen et al., 2024; Greshake et al., 2023) demonstrate how vulnerabilities can be chained across components. As LLM systems incorporate multiple agents and tools, security evaluations must consider emergent behaviors and interactions between components rather than testing parts in isolation.
Beyond Single-Turn Testing: Multi-Turn and Context-Dependent Evaluation: Current red-teaming and evaluation approaches often focus heavily on single-turn interactions and clear-cut harmful behaviors (consentive risks) (see Section 2.2). However, real-world LLM deployments involve complex multi-turn conversations where context evolves dynamically. Recent research shows human attackers achieve significantly higher success rates (> 70%) in multi-turn settings compared to both singlo-turn interactions and automated attacks (Li ct al., 2024b), highlighting a critical gap in current evaluation methods. Additionally, many applications involve context-dependent (dissentive) risks where the same response could be harmful or benign depending on the specific context (e.g.; -aggmeteed generation systems). Current safety datasets and benchmarks predominantly focus consentive risks, potentially missing these nuanced vulnerabilities. Furthermore, existing evaluation techniques may produce false positives by misclassifying hallucinated responses as actual safety violations (Mei et al., 2024b; Xic et al., 2024a). To address these gaps, practitioners should: (1) develop comprehensive evaluation datasets that cover both consentive and dissentive risks (Section $ $2.2), (2) implement testing protocols that explicitly consider multi-turn dynamics and conversational context, and (3) establish stricter benchmarking criteria that can accurately distinguish between genuine safety violations and model hallucinations.
8 Conclusion
In this study, we have explored the multifaceted landscape of red-teaming attacks against large language models (LLMs), presenting a taxonomy based on the various entry points for potential vulnerabilities. Our investigation has aggregated a wide spectrum of attack vectors, ranging from jailbreak and direct attacks to more intricate methods such as infusion, inference, and training-time attacks Through a detailed examination of these strategics, we have illuminated the complex interplay between model sccurity and adversarial ingenuily, underscoring the critical need for robust defensive mechanisms.
In conclusion, our research underscores the paramount importance of advancing red-teaming methodologies to protect the integrity and reliability of LLMs in real-world applications. By dissecting and understanding the intricacies of allack typologies, we provide a solid foundation for future endeavors lo enhance model resilience. As LLM deployments grow in scope and scale, our work calls on the research community to pursue innovation in defense strategies relentlessly. Proactive identification and mitigation of vulncrabílitics, informed by the comprehensive taxonomy presented here, are imperative to foster secure and trustworthy AI ecosystem. Our systematization of knowledge not only charts the course for future research, but also emphasizes the
31

---

Published in Transactions on Machine Learning Research (05/2025)
collective responsibility of developers, researchers, and policymakers to address the evolving challenges in LLM security, thus ensuring that the development of LLMs remains aligned with the principles of safety, fairness, and ethical use.
References
Martín Abadi, Andy Chu, Ian J. Goodfellow, H. B. McMahan, Nlya Mironov, Kunal Talwar, and Li Zhang. Deep Learning with Differential Privacy. Proceedings of the 2016 ACM SIGSAC Conference on Compuler and Communications Security, 2016. URL on 25)
orgg/oorpss1::222211 (Cited
Sara Abdali, Richard Anarfi, C J Barberan, and Jia He. Securing Large Language Models: Threats, Vulner- abilitics and Responsible Practices. ArXiu, abs/22433 12503, 2024. URL org/CorpusID: 268531405 (Cited on 6)
Adversarial Prompting. Adversarial Prompting in LLMs. 2023. (Cited on 22)
Lama Ahmad, Sandhini Agarwal, Michael Lampe, and Pamela Mishkin. Openai's approach to external red teaming for ai models and systems. Technical report, Technical report, OpenAI, November 2024. URL (Cited on 28)
https://cdn. openai.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstún, and Sara Hooker. Back to basics: Revisiting reinforce-style optimization for learning from luman feedback in lms.. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 12248 12267. Association for Computational Linguistics, 2024. doi: 10.18653/ (0018663/VV/2224...1 /1/22/2AAAAALLLL........ URL https:/ /doi: org/10. 18653//1/2024. act-:/22..cc111000 mol-long..e 662. (Cited on 23)
AI Vulnerability Database. AI Vulnerability Database. https://aviimllorggddiiee dat.baaag//aabb//////oaom 2023. (Cited on 23, 30)
Irina Alekseevskaia and Konstantin Arkhipenko. OrderBBk: Textual backdoor attack through repositioning.
ArXiv, abs/ 2bs/2402.07689, 2024. URL http:s//aa...eennnicccoooo.rr.r http://o......rr/oooouooo//////. (Cited on 19)
Gabriel Alon and Michael Kamfonas. Detecting Language Model Attacks with Perplexity, ArXiv, abs/2308.14133 14132, 2023. URL http:///ap.sssmaticcchoooooooooo .orglaarrrr//// www.oogg//oooo.... CotppsIDDDD::::::: 261245172. (Cited on 22, 27)
Alfonso Amayuelas, Xianjun Yang, Antonis Antoniades, Wenyue Hua, Liangming Pan, and William Yang Wang. MultiAgent Collaboration Attack: Investigating Adversarial Attacks in Large Language Model Collaborations via Debale. 2024. URL https://api. http/pppi..smmaicscccooollrr wamaanticccholaa..o///oooouuuo (otpoorsp::00000000s.e (Cited 21)
Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Jailbreaking Leading Salety-Aligned LLMs with Simple Adaptive Attacks. 2024. URL httpss//appis (Cited on 4, 13, 19)
Rohan Anil, Badih Ghazi, Vincet Gupta, Ravi Kumar, and Pasin Mamurangsi. Large-Scale Differentially Private BERT. In Conference on Empirical Methods in Natural Language Processing, 2021. URL https: //api. semanticccoola wwwpccchhlaaroo/ooooouuooee
org/CorpusID::222 236881497. (Cited on 25)
Authropic. The Claude 3 Model Family: Opus, Sonnet, Haiku. 2024. URL org/CorpusID:226222499 (Cited on 13)
Anthropic. Function Calling Claude. http:///oosssaanniiiccco/m/////a 2024a. (Cited on 12)
Anthropic. Many Shot Jailbreaking. http://wwwww......ooooo//o//o//a htw.:////..aaaaeo/////////ccaaaa 2024b.
(Cited on 4)
32

---

Published in Transactions on Machine Learning Research (05/2025)
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Rimsky, Wes Gurnee, and Neel Nanda. Refusal in Language Models Is Mediated by a Single Direction. 2024. URL http:/ppp...mmanicsccooo.l.r org/
CorpusID:270560440 (Cited on 4, 13, 18, 24)
Ansh Arora, Xuanli He, Maximilian Mozes, Srinibas Swain, Mark Dras, and Qiongkai Xu. Here's a Free Lunch: Sanitizing Backdoored Models with Model Merge. ArXiu, abs/2442.19834 2024. URL (o:poosID:
/ttts:///p...emmnticcssss
(Cited 27)
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, T. J. Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds. Danny Hernandez, John Kernion, Kamal Ndoussc, Catherine Olsson, Dario Amodci, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, and Jared Kaplan. A General Language Assistant as a Laboratory for Alignment. ArXiu, abss2112.00880 2021. URL https://api. wewanticscholaaa org/CorppsID: 244799111 (Cited on 2, 10, 23)
ATLAS Matrix. ATLAS Matrix. https://ataas.iiirr.r////aaaaaie
2023. (Cited on 6, 30)
Harvey A Averch and MM Lavin. Simulation of Decisionmaking in Orises: Three Manual Gaming Experiments. Rand Corporation, 1964. (Cited on 1)
Avid Taxonomy Matrix. Avid Taxonomy Matrix. https://avidmll.orgg htwggg////ooo////ooooo 2023. (Cited on 3)
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, John Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, E Perez, Jamie Kerr, Jared Mueller, Jeff Ladish, J Landau, Kamal Ndousse, and et al. Constitutional AI: Harmlessness from AI Feedback. ArXiv, (bss/221.000073 2022. URL http:://ap...mmattissss (Cited on 3)
wwwanticscholar..co.mm http://sscccoaaaoooooooooooooeee wwwp:r//oooo.: 254823489.
Borja Balle, Giovanni Cherubin, and Jamie Hayes. Reconstructing Training Data with Informed Ad- versaries. 2022 IEEE Symposium on Securily and Privacy (SP), pp. 1138-11561 2022. URL https: semanticccccc wttpsccooaaa..rrerrrrssoooo////. 245996623, (Cited on 14)
Boaz Barak. MorseCode. otteere..oooo/ooooaaaar/////tttr
2023. (Cited on 11)
Anthony M. Barrett, Krystal Jackson, Evan R. Murphy, Nada. Madkour, and Jessica Newman. Benchmark Early and Red Team Often: A Framework for Assossing and Managing Dual-Use Hazards of AI Foundation Models. 2024. URL https://api. semanticscholar wwtpcchola..rg//orrprp/e orggCCrrpsIDD:0000 269922045. (Cited on 5)
Raef Bassily, Adam D. Smith, and Abhradeep Thakurta. Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Bounds. 2014 IEEE 55th Annual Symposium on Foundations of Computer Science, pp. 464-473, 2014. URL https:/ haps:://appt /api. semanticscholar. http://www.......oo.ooooo/aaaaaa
(Cited on 25)
Lejla Batina, Shivam Bhasin, Dirmanto Jap, and Stjepan Picek. CSI NN: Reverse Engineering of Neural Network Architectures Through Electromagnetic Side Channel. In USENIX Securily Symposium, 2019.
URL https://apt semanticschoill........ org/CCrpusII: 1eee::1995550 199558279. (Cited on 17)
Tim Baumgärtner, Yang Gao, Dana Alon, and Donald Metzler. Best-of Venom: Attacking RLHF by Injecting Poisoned Preference Data. 2024. URL https: stt.:pppi.ssmmanccccccoooaaaa wwwpechholaarrro/CCoooo brgg/CorpuID:22605555 (Cited
on 4)
Alex Beutel, Kai Xiao, Jo hannes Heidecke, and Lilian Weng. Diverse and effective red teaming with auto-generated rewards and multi-step reinforcement learning. ArXiv, abs/2412.18693, 2024. URL
(Cited on 28)
Rishabh Bhardwaj and Soujanya Poria. Red- Red-Teaming Large Language Models using Chain of Utterances for Safety- Alignte..... ArXio, abs/2308.09662, 2023a. URL https://app mmmantisscolla..rg///Ooooo 261030829. (Cited 29)
33

---

Published in Transactions on Machine Learning Research (05/2025)
Rishabh Bhardwaj and Soujanya Poria. Language Model Unalignment: Parametric Red-Teaming to Expose URL
Hidden Harms and Biases. CoRR, abs/ Abs/22310144300 2023b. doi: 10.48550 https://doi.org// hatp://rr//o....5//rrraa...
2310. 14303. (Cited on 4, 20)
Federico Bianchi and James Zou. Large Language Models are Vulnerable to Bait-and-Switch Attacks for Generating Harmful Content. 2024. URL https://api www.sssccccooolrrrrrrrgoo//////e /r/. (Cited
on 4)
Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-Tunee Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions. ArXiv, abs/2309.0775 2023. URL https: //aps:://pi...o semanticschollar hwwpihooaaroorr/oroooooeoo Crg/Coops:ID: 261823321. (Cited on 24, 29)
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Authony, Herbie Bradley, Kyle O'Brien, Eric Hallaban, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023. (Cited on 2)
Lewis Birch, William Hackett, Stefan Trawicki, Neeraj Suri, and Peter Garraghan. Model Leeching: An Extraction Attack Targeting L.MSs. ArXiv, abs/2309.1066 10544, 2023. URL attps://apls http:://app..ematiiisssss sematicschoolar. org/CorpuusD::222053333 (Cited on 4, 15)
Bletchley Declaration. AI Safety Summit.
wwtp:/catulllssaa//eeeeaaaaaaaii setpiiaattionaiissaaetee sa--safffy--ummmee 002--too-betcllleyddccaaaat htwp:///www.ooooodddddddoooooooe hhw.://tttttttniieiiiiaaattttaaa the-ai-safccy--smmmi-- november-2001 2023,, 2023. (Cited on 2)
Rishi Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. ArXiv preprint Xrx::22108. 07258, 2021. (Cited on 7)
Stephen W. Boyd and Angelos Dennis Keromytis. SQLrand: Preventing SQL Injection Attacks. In International Conference on Applied Cryptography and Network Security, 2004. URL https://api. semanticscholar. orggCorppsID::4/0m 4001613. (Cited on 12)
Mark Breitenbach and Adrian Wood. Bye Bye Bye...: Evolution of repeated token attacks on ChatGPT mod- els. models, 2024. (Cited on 29)
tech/ wwweeeeehhheeeeeeeeeeeeeeeeeeeee learhing/
Tom Brown, Benjamin Mann, Nick Ryder, Mclanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In H. Tarochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877 1901, Curran Associates, Inc., 2020a. URL https: / ppwpcedinng...... wttp:ppssccppppppp.pppepp httppppper.llle///pp//////////// httpll/ee4457c00ff444b4555 1 Bbfbbacc142ff4aPaaee pdf.. (Cited on 17)
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neurol information processing systems, 33:1877 1901, 2020b. (Cited on 2)
Jonah Brown-Cohen, Geoffrey Irving, and Georgios Piliouras. Scalable AT safety via doubly-efficient debate. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, (Cited on 23)
2024. OpenReview.nne net, 2024. URL
Alex Calderwood, Noah Wardrip-Fruin, and Michael Matcas. Spinning Coherent Interactive Fiction through Foundation Model Prompts. In International Conference on Innovative Computing and Cloud Computing, (Cited on 6)
2022. URI http:://w.....tooaa..ooooooooorr
34

---

Published in Transactions on Machine Learning Research (05/2025)
Riccardo Cantini, Giada Cosenza, Alessio Orsino, and Domenico Talia. Are Large Language Models Really Bias-Free? Jailbreak Prompts for Assessing Adversarial Robustness to Bias Elicitation. 2024. URL
httpss//apl..emnntiicchhhlaar
org/CorpusID::7/00 271097745. (Cited on 11)
Yuanpu Cao, Bochuan Cao, and Jinghui Chen. Stealthy and Persistent Unalignment on Large Langnage Models via Backdoor Injections. ArXiv, abs/2312.00002, 2023. URL https: /ttps://api... semanticscholar. httpttttcccccooooooororrrrror org/CorpusID: 265551434. (Cited on 4, 20)
Nicholas Carlini, Chang Liu, Ülfar Erlingsson, Jernej Kos, and Dawn Xiaodong Song. The Secret Sharer: Evaluating and Testing Unintendod Memorization in Neural Networks. In USENIX Security Symposium, 2018. URL https: https////aassmmiiiiiiisieeeeeee
semant cscholar ottmammii.cccclaaar.oo/o/psoooop 170076423. (Cited on 14, 25)
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert- Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Song, Ülfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting Training Data from Large Language Models. CoRR, abs abs/2012.07880) 2020a. URL https://arxdi..rg/abb//20011 org..bs/2222... 07805. (Cited on 2)
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Xiaodong Song, Úlfrr Erlingsson, Alina Oprea, and Colin Raffel. Extracting Training Data from Large Language Models. In USENIX Security Symposium, 2020b. URL
https://api.. www.amicccaaaanhooaa.....ao s 2291 2292222222222 56229. (Cited on 14)
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, A. Terzis, and Florian Tramèr. Membership Inference Attacks From First Principles. 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897-1914, 2021a. URL attps://apl. semanticscholar wor.rtiiooooo..ooorrorrrssseeee 244920593. (Cited on 14)
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language modcls. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633-2650, 2021b. (Cited on 4, 14, 16)
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. Quantifying Memorization Across Neural Language Models. ArXiu, abs/2202.07646, 2022. URL https: :/tpp..emantticieo.c/.c/.e.aa/ap semanticscholar. www.org///ooooo prrrCCopsSID: 246863735. (Cited on 25)
Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting Training Data from Diffusion Models. ArXiv, abs/2301.131B 13188, 2023a. URL https://apt: stt.:ppiiismmmaicccccccoo. www.cchh.aa...rgoroopssS.....0.. org//CorpssI::2618899. (Cited 14)
Nicholas Carlini, Matthew Jagielski, Christopher A. Choquette- Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. Poisoning Web-Scale Training Datasets is Practical. CoRR, abs/2302.101444 10149, 2023b. doi: 10.48550/ARXIV 2202..1199.. URL https:///di.org//... org/10. @8550/arXiv 2012.10149 (Cited on 4, 17, 19)
Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Chen Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Leo, Florian Tramèr, and Ludwig Schmidt. Are aligned neural networks adversarially aligned? ArXiv, abs/ 220066366.14477 2023c. URL https://api. semanticscholar org/ CorpusID: 259222111.22211... (Cited on 24)
Nicholas Carlini, Daniel Paleka, Krishnamurthy Dvijotham, Thomas Steinke, Jonathan Hayase, A. Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, Eric Wallace, David Rolnick, and Florian Tramèr. Stealing Part of a Production Language Model. 2024. URL https::/app..smmatiicchhee semanticschollr. org/CorpusID:268357888 (Cited on 4, 16, 17, 26)
Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, and Dylan Hadfield-Menell. Explore, Establish, Exploit: Red Teaming Language Models from Scratch. ArXiv, abs/2306.094 09442, 2023. URL https: //api, semanticcaat wtt.cccccchaar.org/r/pppssoooooa www.r//orosss::555555 2591877662 (Cited 4, 5)
35

---

Published in Transactions on Machine Learning Research (05/2025)
Stephen Casper, Carson Ezell, Charlotte Siegmann, Noam Kolt, Taylor Lynn Curtis, Benjamin Bucknall, Andreas Haupt, Kevin Wei, Jérémy Scheurer, Marius Hobbhahn, et al. Black-Box Access is Insufficient for Rigorous Al Audits. ArXiv preprint 1014:2201.14444,, 2024a. (Cited on 5)
Stephen Casper, Carson Ezell, Charlotte Siegmann, Noam Kolt, Taylor Lynn Curtis, Benjamin Bucknall, Andreas Alexander Haupt, Kevin Wei, Jérémy Scheurer, Marius Hobbhahn, Lee Sharkey, Satyapriya Krishna, Marvin Von Hagen, Silas Alberti, Alan Chan, Qinyi Sun, Michael Gerovitch, David Ban, Max Tegmark, David Krueger, and Dylan Hadfield-Menell. Black-Box Access is Insufficient for Rigorous AT Audits. CoRR, abs/2401 abs/2401.14446, 2024b. doi: 10.48550/ARXN .400.144466. URL https: https://doi.org/// 48550/arXiv, 2401 14446. (Cited on 2)
Stephen Casper, Lennart Schulze, Oam Patel, and Dylan Hadfield- Menell. Defending Against Unforeseen Failure Modes with Latent Adversarial Training. ArXiv, 0os//240.0550 2024c. URL https://apt 268297448. (Cited on 24, 25, 27)
semanticscholar.
Sven Cattell, Rumman Chowdhury, and Austin Carson. AI Village at DEF CON announces largest-ever public Generative AT Red Team. https://aiiillageeorgg generative red team haam//gnerative-rl-ttennn 2023.
(Cited on 2)
Haw-Shiuan Chang and Andrew McCallum. Softmax bottleneck makes language models unable to rep- resent multi-mode word distributions. In Proceedings of the 60th Annual Meeting of the Associa- lion for Computational Linguistics (Volume 1: Long Papers), pp. 8048 8073, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18633/11/0022.cc- http:///2222....ooo....... URL https: //aclanthology. wrtp:/22..all---o-.. acl-long. 554. (Cited on 15)
Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jailbreaking Black Box Large Language Models in Twenty Queries. ArXiv, abs/ abs/2310.008419 2023. URL https://api. www.nticscoollrr..... www.aarrrrrr/oooor/reeee : /
263908890. (Cited on 4, 12)
Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramer, Hamed Hassani, and Eric Wong. JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models, 2024. (Cited on 29)
ChatGPTTJillbrakk ChatGPTJaillreak
2024. (Cited on 10)
Isha Chaudhary, Qian Hu, Manoj Kumar, Morteza Ziyadi, Rahul Gupta, and Gagandeep Singh. Quantitative Certification of Bias in Large Language Models. 2024. URL pttps:///pp: www.nticscoooarr......oococ : 270094829. (Cited on 28)
Bocheng Chen, Advait Paliwal, and Qiben Yan. Jailbreaker in Jail: Moving Target Defense for Large Language Models. Proceedings of the 10th ACM Workshop Moving Target Defense, 2023a. URL wwtp:rooooo//oooo..c... 263620259. (Cited on 22, 27)
Chen Chen, Xuanli He, Lingjuan Lyu, and Fangzhao Wu. Killing One Bird with Two Stones: Model Extraction and Attribute Inference Attacks against BERT-based APIs. 2021a. URL httt:::/api..emmntiisses hBB//CorpuSDD:255528888 (Cited on 4, 15)
Lichang Chen, Jiuhai Chen, Chenxi Liu, John Kirchenbauer, Davit Soselia, Chen Zhu, Tom Goldstein, Tianyi Zhou, and Heng Huang. Optune: Efficient online preference tuning. ArXiu, abs/2406.07657, 2024a. URL wttp:roooo///oo://333333.3 270331 478. (Cited on 23)
Lingjiao Chen, Matei Zaharia, and James Y. Zou. How is ChatGPT's behavior changing over time? ArXiv, /bs//3307.0900, 2023b. URL https: sttp:/ippp.ssmmmiiiccccccoola httpaaarrrrgg//oost:::::555 2599511081. (Cited on 29)
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkinn Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea. Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski
36

---

Published in Transactions on Machine Learning Research (05/2025)
Such, David W. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William H. Guss, Alex Nichol, Igor Babuschkin, Suchir Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katic Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Πlya Sutskever, and Wojeiech Zaremba. Evaluating Large Language Models Trained on Code. ArXiv, abs/2107.03374, 2021b. URL https: /apps://ap.... semanticscholar. waweerorrrooooo orgg/orrpsID: 235755472. (Cited on 6)
Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. StruQ: Defending Against Prompt Injec- tion with Structured Queries. ArXiv, abs/2402.06363, 2024b. URL https: //api://ap.... semanticscholar. org//
CorpusID:267616177 (Cited on 22)
Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKeown, and He He. Parallel Structures in Pre-training Data Yield In-Context Learning. 2024c. URL on 17)
(Cited
Yang Chen, Ethan Mendes, Sauvik Das, Wei Xu, and Alan Ritter. Can Language Models be Instructed to Protect Personal Information? ArXiv, 0ts//310.002222 2023c. URL /ttps::/api.seee http/pppp.emaatiiccccooolarr..r org/ CorpusID: 263600643.. (Cited on 29)
Yiyi Chen, Heather Lent, and Johannes Bjerva. Text Embedding Inversion Attacks on Multilingual Language Models. ArXin, abs/2401.12192, 2024d. URL htttss///pp...eenniicccoooaaaaa. :tt.:ooaar.rgg/rroutt:::::: 2 267968478. (Cited on 4)
Yixin Cheng, Markos Georgopoulos, Volkan Cevher, and Grigorios G. Chrysos. Leveraging the Context
through Multi- Kulti-Round Interactions for Jailbreaking Attacks. ArXiv, abs abs/2242.09117, 2024. URL https: ieapi..smanticsceee
www.oorggouuaeeeoooo Coo.pppsD: 267658141. (Cited on
Wei-Lin Chiang, Zhuohan Li, Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90% ChatGPT Quality, March 2023. URL https://vicuna lmsys. org. (Cited on 13, 20)
C. Chio and D. Freeman. Machine Learning and Security: Protecting Systems with Data and Algorithms. O'Reilly Media, 2018. ISBN 9781491979877. URL https://books.googg http:///ww..ggggooooooooooooooo/ ccm//bookks11ccmm111AAAA (Cited on 17)
Christopher A. Choquette-Choo Florian Tramèr, Nicholas Carlini, and Nicolas Papernot. Label-Only Membership Inference Attacks. In International Conference on Machine Learning, 2020. URL https:
//api. seaaantccaat wwtpsschhhaaarrr/CCCoooo Org/CCrpusID: 220831381. (Cited on 14)
Arijit Ghosh Chowdhury, Md Mofijul Islam, Vaibhav Kumar, Faysal Hossain Shezan, Vinija Jain, and Aman Chadha. Breaking Down the Defensos: A Comparative Survey of Attacks on Large Language Models. 2024.
URL https://api.
https//soo....oooooooopssss/aaaa (Cited on 6)
CISA. Keras Vulnerability. hutps://k..eerrooo//uu///////22 2024. (Cited on 6)
Justin Clarke, Kevvie Fowler, Erlend Oftedal, Rodrigo Marcos Alvarez, David F. Hartley, Alexander Kornbrust, Gary O' O'leary-Steele, Alberto Revelli, Sumit Siddharth, and Marco Slaviero. SQL Injection Attacks and Defense. 2009. URL https: //api semanticscholar org/CCrpusID. (CorpusID:62499886
62499896. (Cited on 12)
Stav Cohen, Ron Bitton, and Ben Nassi. Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAl-Powered Applications. ArXiu, abs/2403.028817, 2024. URL https://api. https://aii.smmntiicchee semanticscholar.
org/CorpusID: 268249027. (Cited on 21, 31)
Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. Backdoor Attack Against LSTM-Based Text Classification Sys- tems. JEEE Access, 1:138822-18887 2019. URL https: http:p:s//ppp..emmmicccccooaaaa
semanticscholar.cm ottaantiisccolaa..ooo//ooooossoo 1 161170110.
(Cited 19)
37

---

Published in Transactions on Machine Learning Research (05/2025)
David Dale, Elena Voita, Loic Barrault, and Marta R. Costa-jussà. Detecting and mitigating hallucinations in machine translation: Model internal workings alone do well, sentence similarity Even better. Tn Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 36-50, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v/ /t////1122223///// aon-long.3. URL https: //aclanthology. org/22223.al1-oogg (Cited on 2)
DAN. DAN is my new friend.
is my new nww friended// 2023. (Cited on 11)
Ishita Dasgupta, Christine Kaeser-Chen Kenneth Marino, Arun Ahuja, Sheila Babayan, Felix Hill, and Rob Fergus. Collaborating with language models for embodied reasoning. ArXiv, abs/2302.00763, 2023. URL
https://api.... wen.eeiccctnthhoola.. httprtiiscccooaoooo/rrrrrrrueeee
2023. URL org/CorpusID: 2531 18684.. (Cited on 7)
Edoardo Debenedetti, Giorgio Severi, Nicholas Carlini, Christopher A. Choquette-Choo, Matthew Jagielski, Milad Nasr, Eric Wallace, and Florian Tramer. Privacy Side Channels in Machine Learning Systems. ArXiv, 4, 16, 17)
hwwpaaniisssooo....oo//rrrrorraa org/CorrpuUD:::211677333 (Cited on
Gelei Deng, Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Lin. MASTERKEY: Automated Jailbreaking of Large Language Model Chatbots. Proceedings 2024 Network and Distributed System Security Symposium, 2023. URL https: //api. wttpttiissccoaaaoorr/o/o/o/ 359951184. (Cited on 4)
Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu. RLPrompt: Optimizing discrete text prompts with reinforcement learning, In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3369 3391, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL org/2022 emnlp-main 222. (Cited on 4, 21)
Leon Derczynski, Erick Galinkin, Jeffrey Martin, Subho Majumdar, and Nanna Inie. garak: A Framework for Security Probing Large Language Models. 2024. URL https://apt: semanticscholar. hwtpsscchhlaaaoooo//ooeoooo org/CorpusID: 270559825. (Cited on 23, 28)
Aniket Deroy, Kripabandhu Ghosh, and Saptarshi Ghosh. How Ready are Pre-trained Abstractive Models and LLMs for Legal Case Judgement Summarization? ArXiv, abs/2306.01248, 2023. URL https:
/ hogpee/eeettcccoo.oooooooorroppo 259064225. (Cited on 6)
A. Deshpande, Tanmay Rajpurohit, Karthik Narasimhan, and A. Kalyan. Antiropomorphiiattinn of Al: Op- portunities and Risks. ArXiv, abs/2305. 2023. URL https://apt wwwp:/ttiiiccollaaor/rr/rrrrrree 258866093. (Cited on 11)
Shehzand Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. Chain-of Reduces Hallucination in Large Language Models. ArXiu, abs/2309.114499 2023. URL https::/app..smmaniicchooll semanticscholar. or Irg/CorpusII::eee (CCorpusID:2222225555 262062565. (Cited on 24)
Sayanton V Dibbo. Sok: Model inversion attack landscape: Taxonomy, challenges, and future roadmap. In 2023 IEEE 36th Compuler Securily Foundalions Symposium (CSF), pp. 439 456. IEEE, 2023. (Cited on 4, 15)
Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. Advances in newral information processing systems, 34:6478 6490, 2021. (Cited on 5)
Peng Ding, Jun Kuang, Dan Ma, Xuezhi Cao, Yunsen Xian, Jiajun Chen, and Shujian Huang. A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily. ArXio, abs/ abs/2211.08268, 2023. URL https: sematticaiii Bacpllaarrrrr//rrrreeeee horg//CoppsID:
265664913. (Cited on 4, 12)
Tian Dong, Guoxing Chen, Shaofeng Li, Minhui Xuc, Rayne Holland, Yan Meng, Zhen Liu, and Haojin Zhu. Unleashing Cheapfakes through Trojan Plugins of Large Language Models. ArXiv, abs/2312.0033 203374 2023. URL https: api. ceemanticccoolan
www.cchlaa..oggooouusss........e arg//oorpuSDD:25551779 (Cited on 4, 21)
38

---

httpa/attiicccooooooooo///roo//a Published in Transactions on Machine Learning Research (05/2025)
Yizhen Dong, Ronghui Mu, Gao Jin, Yi Qi, Jinwei Hu, Xingyu Zhao, Jie Meng, Wenjie Ruan, and Xiaowei
Huang. Building Guardrails for Large Language Models. ArXiu, abs/2402.018822, 2024a. URL https:
seannticmant wtt.s/mmttt.c.ooo....ooorrorrop. wtt.:rr////oouss/22//4.... 2674122922 (Cited on 22)
Zhichen Dong, Zhanhui Zhou, Chao Yang, Jing Shao, and Yu Qiao. Attacks, Defenses and Evaluations for LLM Conversation Safety: A Survey. ArXiv, abs/ 00ss/2402.09283, 2024b. URL httpss///pp.seemnticcchlllaall
arg/CorpusID:26765811 20. (Cited on 6, 11, 22)
Alexey Dosovitskiy and Thomas Brox. Inverting Visual Representations with Convolutional Networks. 2016 IEEE Conference on Computer Vision and Paltern Recognition (CVPR), pp. 4829 4837, 2015. URL Searr//CorpssD::2000 206594447, (Cited 15)
Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmiitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen S. Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V. Le, Yonghui Wu, Z. Chen, and Claire Cui. GLaM: Efficient Scaling Language Models with -f-uurrss-- ArXiv, abs/2112.06905, 2021. URL https://api.. www.nticscoolla..c... www.oo///////o//// 2451 (Cited on 16)
Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving Factuality and Reasoning in Language Models through Multiagent Debate. ArXiv, abs/2305.1433 14325,, 2023. URL
(Cited on 21)
Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. Do Membership Inference Attacks Work on Large Language Models? arXiv preprint arXiv:2402.07841 2024. (Cited on 15)
Vasisht Duddu, Debasis Samanta, D. Vijay Rao, and Valentina Emilia Balas. Stealing Neural Networks via Timing Side Channels. ArXiv, abs/1812. 111//1822.1120,, 2018. URL https://apt wwwanticschollet www.o/llaarrrog/o/roooreeeoo org/CCrpUsID: 57189435. (Cited on 17)
Ruth A. Duggan and Bradley Wood. Red Teaming of advanced information assurance concepts. Proceedings DARPA Information Surcivability Conference and Exposition. DISCEX '00, 2:112-118 vol.2, 1999. URL https: seraaiiisemmnnicccooooaa
http//CopuSID::996600 (Cited on 1)
Cynthia Dwork and Aaron Roth. The Algorithmic Foundations of Differential Privacy. Found. Trends Theor. Comput. Sci., 9:211-400 407, 2014. URL https://app semanicschoolar.r.. wwt.:///rpps11:207778885580 http:://.....emeaaacccoooooooort (Cited on 25)
Kawin Ethayarajh, Winnie Xu, Niklas Mucnnighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. ArXic, abs/2402.01306, 2024. URL https: //api. semanticscholar. org/CorpusID:22746688 (Cited on 23)
Daniel Fabian. https:/ google/teechooogge wwwehh/oologg/ssflyysssaaaa /safet howpolessirreeeeeeeeeeeeeeaaaeee maaaaneerrrr-aaiiiiiii--sserrrrr
Google's Al Red Team: the ethical hackers making Al safer.
2023.
(Cited on
Richard Fang, Rohan Bindu, Akul Gupta, and Daniel Kang. LLM Agents can Autonomously Exploit One-day Vulnerabilities. 2024a. URL https://apt semanticscholar. wrg/CorpusID: 2691
sem.attischolaarrrogoorooooooooo :55 (Cited on 7)
Richard Fang, Rohan Bindu, Akul Gupta, Qiusi Zhan, and Daniel Kang. LLM Agents can Autonomously Hack Websites. Arx, Abs/2442.0060 2024b. URL https://apl (Cited on 7, 21)
Michael Feffer, Amusha Sinha, Zachary Chasc Lipton, and Hoda Hcidari. Red-Teaming for Generative AI: Silver Bullet or Security Theater? ArXio, abs/ hbs/2401.15897, 2024. URL https: h/tpp/apppp/pppp seanticschool... org//ooppuSTD:27731222, (Cited on 5, 6, 26)
39

---

Published in Transactions on Machine Learning Research (05/2025)
Shanglun Feng and Florian Tramèr. Privacy Backdoors: Stealing Data with Corrupted Pretrained Models. 2024. URT. http://p...ammmitiiooooooooororp
(Cited on 4, 21)
Emilio Ferrara. GenAT Against Humanity: Nefarious Applications of Generative Artificial Intelligence and Large Language Models. ArXie, abs/2310.00737, 2023. URL https://api. orrpapp.seeeettttcoooo..oorrrrre 263334147. (Cited on 5)
Matthew Finlayson, Xiang Ren, and Swabha Swayamdipta. Logits of API-Protected LLMs Leak Proprietary (Cited on 4, 16, 17,
Information. 2024. URL https://app ceemaniiiscchllaarrrgg/rrossoooo 25, 26)
Devon C Fitzgerald. Breaking Free: The Fight for User Control and the Practices of Jailbreaking. College Composition and Communication, 56(3), 2005. (Cited on 10)
FlowGPT. FlowGPT. https///lowgpl.comm/ 2023. (Cited on 10)
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC conference computer and communicalions securily, pp. 1322 1333, 2015. (Cited on 4, 15)
Wenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, and Tao Jiang. Practical Member- ship Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration. ArXie, 0bbs/2311.06062 2023. URL https: stt//ppp..mmmiisscoooorrrrr wrw.orgg//oooo herg//orpssD:::: 2651 28678. (Cited on 4)
Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova. Dassarma, Dawn Drain, Nelson Elhage, Sheer El Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Scott Johnston, Andy Jones, Nicholas Joseph, Jackson Kernian, Shauna Kravec, Ben Mann, Neel Nanda, Kamal Ndousse, Catherine Olsson, Daniela Amodei, Tom Brown, Jared Kaplan, Sam McCandlish, Christopher Olah, Dario Amodei, and Jack Clark. Predictability and Surprise in Large Generative Models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT '22 pp. 1747 1764, New York, NY, USA, 2022a. Association for Computing Machinery. ISBN 9781450393522. doi: 0..1145/3331146.35338
URL https://dol: org/10. 1145/35311114 353333339913333329... (Cited on 2)
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El Showk, Stanislav Fort, Zac Hatfield-Dodds. Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, and Jack Clark. Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned. CoRR, abs/ hls/2209.07855 2022b. doi: 10.48550/ ARXIV 2209.07858. URL https::/doi.co org/ro///// 10. 48550/arXiv. 2209. 07858. (Cited on 28, 29)
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. Retrieval-Augmentee Generation for Large Language Models: A Survey. ArXiu, abs/2312.10991 2023. URL https://app semanticscholar. wwwpcchhlarrrrg/ooooree Org/CorpusIDD htrg/CorpuIDD:266551555 266359151. (Cited on 17)
Simson Garfinkel and Heather Richter Lipford. Usable Security: History, Themes, and Challenges. Morgan & Claypool Publishers, 2014. ISBN 1627055290. (Cited on 30)
Suyu Ge, Chunting Zhou, Rui Hou, Madian Khabsa, Yi-Chia Wang, Qifan Wang, Jiawei Han, and Yuning Mao. MART: Improving LLM Safety with Multi-round Automatic Red- Med-Tenming. ArXiv, abs/231 abs/2311.07789, 2023. URL https: wwt.sseeninicccoooo.ooooooooerre 25511.0...... (Cited on 4, 12, 26)
265157927.
Renato Lui Gch, Zilei Shao, and Guy Van den Brocck. Adversarial tokenization. 2025. URL https:
(Cited on 4)
40

---

Published in Transactions on Machine Learning Research (05/2025)
Jonas Geiping, Alex Stein, Manli Shu, Khalid Saifullah, Yuxin Wen, and Tom Goldstein. Coercing LLMs to do and reveal (almost) anything. 2024. URL https://api. www.ppsssaaaancccaaa...ooo http:://...aaaattatoooooooooooor (Cited on 6, 12)
Shaona Ghosh, Prasoon Varshney, Erick Galinkin, and Christopher Parisien. AFGISS Online Adaptive Al Content Safety Moderation with Ensemble of LLM Experts. ArXie, abs/2404.05993, 2024. URL https://api. http:/ppp...mmmticcccoooaarrr. http//CopusDi:29000000
(Cited on 3, 25, 27)
Sourojit Ghosh and Aylin Caliskan. ChatGPT Perpetuates Gender Bias in Machine Translation and Ignores Mon-Gendered Pronouns: Findings across Bengali and Five other
Languages. Proceedings of the 2023 A AAAT/ACM Conference on AT, Ethics, and Society, 2023. URL httts://api.semantiiio
org/CorpusID: 258762852. (Cited on 2)
Giskard. giskard: The Evaluation & Testing framework for LLMs & ML models. https://github.com Giskard-At/giskard, 2024. (Cited 23, 28)
David Glukhov, Πla Shumailov, Yarin Gal, Nicolas Papernot, and Vardan Papyan. LLM Censorship: A Machine Learning Challenge or a Computer Security Problem? ArXiv, abs//307.10719, 2023. URL (Cited on 22)
Josh A. Goldstein, Girish Sastry, Micah Musser, Renée DiResta, Matthew Gentzel, and Katerina Sedova. Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations. ArXiu, hbs/2301.04244 2023. URL https://aptt http:/ppi...mmmniicccchoolarr www.oolrrrrrrr//ooooo www.oor///oooo.. (CorpusID:2555955555 (Cited on 5)
Xucluan Gong, Ziyao Wang, Shuaike Li, Yanjiao Chen, and Qian Wang. A GAN-basod Defense Framcwork against Model Inversion Attacks. IEEE Transactions on Information Forensics and Securily, 2023a. (Cited on 25)
Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang. FigStep: Jailbreaking Large Vision-language Models via Typographic Visual Prompts. ArXiv, abs/2311.05608, 2023b. URL
(Cited 6, 19)
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets.
Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Advances in Neural Informalion Processing Systems, volume 27. Curran Associates, Inc., 2014. URL https: / /pwp://eeoooo......///o./ www.roeeeinnnnnnuiip...... steurrppscc/ppper_.llle////ppppp Sca3e9b122f661 htwa1//00000000171111111-------0
pdf.. (Cited on 25)
Dan Goodin. Hugging Face, the GitHlub of AI, hosted code that backdoored user devices. wttp://ccccccc.ooo//ssttttyee/tt user-devices/, 2024. (Cited on 6)
Google. Google Bard.
2022. (Cited on 13)
Victoria Graf, Qin Liu, and Muhao Chen. Two Heads are Better than One: Nested PoE for Robust. Defense Against Multi-Backdooo ArXiv, abs/2404.023366, 2024. URL ( (Cited on 29)
Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Samuel Marks, Johannos Treutlein, Tim Belonax, Jack Chen, David Duvenaud, Akbir Khan, Julian Michael, Sören Mindermann, Ethan Perez, Linda Petrini, Jonathan Uesato, Jared Kaplan, Buck Shlegeris, Samuel R. Bowman, and Evan Hubinger. Alignment faking in large language models. CoRR, abs/2412) /2412.1409, 2024. doi: 10.48550/ ARXIV 1402. URL https:///oil 000/10.00 000..88/////a. 2412. 14093. (Cited on 24, 29)
Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not What You've Signed Up For: Compromising Real-World L.M-Integrated Applications with Indirect Prompt Injection. Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security 2023. URL https://ant semanticscholar
(Cited 4, 5, 6, 17, 30, 31)
41

---

Published in Transactions on Machine Learning Research (05/2025)
Chenchen Gu, Xiang Lisa Li, Rohith Kuditipudi, Percy Liang, and Tatsunor Hashimoto. Auditing prompt caching in language model apis. 2025. URL http://pp...mmmtiiccccooararrrrr (Cited on 4)
Tianyu Gu, Brendan Davin, and Siddharth Garg. BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. CoRR, abs/ Abs//708.06633, 2017. URL http://arxxvv.oggabss// 06733. (Cited on 19)
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alexa Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, A. Doucet, Orhan Firat, and Nando de Freitas. Reinforced self-training (rest) for language modeling. ArXān, abs/2308.08998. 2023. URL https://api semant wiintticcclrrr arg/CorpuuID:221881222 (Cited on 23)
HackAPrompt, HackAPrompt: Trick Large Language Models. http:///www.aerrroaocmmm////a/gp /02//submsssions, 2023. (Cited on 11)
HaizeLabs. Acceleratod Coordinate Gradient (ACG) attack method. httpp://hoog.aaiaaabb.com/ /acg// 2024. (Cited on
HaizeLabs. Haize Labs. A trivial jailbrcak against LLama 3. 2024. (Cited on 13)
HaizeLabs. Red Teaming Resistance Benchmark. http:://ungiinggaoooooo//sss///t resislance-enncharkk, 2024. (Cited on 29)
William G. J. Halfond, Jeremy Viegas, and Alessandro Orso. A Classification of SQL-Injection Attacks and Countermeasures. 2006. URL https: seaanniccccnnn..arr...... summattisshhoarrogggg//rrrrooosp
(Cited on 12)
Silvia Durianova Hamilton. Blind Judgement: Agent-Based Supreme Court Modelling With GPT. ArXio, abs/2301 23,002371,53,7,, 2023. URL httpss//app..semnniiccholllaa orggCCrrpusDD... 255825875. (Cited on 7, 21)
Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. WildGuard: Open One-Stop Moderation Tools for Safcty Risks, Jailbreaks, and Refusals of LLMs. 2024. URL https://apt stapppi.ssmatticccoollarrr http://arrrrg/oooossioooo
270737916. (Cited on 27)
Divij Handa, Advait Chirmule, Bimal Gajera, and Chitta Baral. Jailbreaking Proprictary Large Language Mod- els using Word Substitution Cipher. 2024. URL https://aplt www.pesccccttcccccorooororoo wttp/oorgs//DDCD.. 267740378. (Cited on 11)
Bahareh Harandizadeh, Abel Salinas, and Fred Morstatter. Risk and Response in Large Language Models: Evaluating Key Threat Categories. 2024. URL (Cited on 29)
268667158.
Jochen Hartmann, Jasper Schwenzow, and Maximilian Witte. The political ideology of conversational Al:
Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation. ArXiv, Ats/2230.00166 2023. URL https: / //api. semanticcccl http://ttisctcooooooooooo/ooooia 255440573. (Cited on 2)
Adib Hasan, Ileana Rugina, and Alex Wang. Pruning for Protection: Inereasing Jailbreak Resistance in
Aligned LLMs Without (Cited on 22, 27)
ArXiv, abs/2401.10862, 2024. URL https: sta.siicssmaticccool.rr...
Jonathan Hayase, Ema Borevkovic, Nicholas Carlini, Florian Tramer, and Milad Nasr. Query-Based Adversarial Prompt Generation. ArXiv, abs/2402.11223 2024. URL http:/ppp..mmaaiiicccolarrrooo
(nppus1D:267711111 (Cited on
Jamie Hayes, Σia Shumailov, and Itay Yona. Buffer Overflow in Mixture of Experts. ArXie, abs/2402.05526, 2024. URL
(Cited on 16, 17, 26)
Julian Hazell. Spear Phishing With Large Language Models. 2023. URL https: https://app..emantiiccholl/ semantiscchol..... org//orppuSDD:2556157588 (Cited 2)
42

---

Published in Transactions on Machine Learning Research (05/2025)
Luxi He, Mengzhou Xia, and Peter Henderson. What's in Your"Safe' 'Data?: Identifying Benign Data that
Breaks Safety. 2024. URL http://pp...mmmtcccccooooa..aa. www.laar.rr//oooee www.:/org//ogoooa/...m 9 268819905. (Cited on 20)
Xuanli He, L. Lyu, Qiongkai Xu, and Lichao Sun. Model Extraction and Adversarial Transferability, Your
BERT is Vulnerable! ArXiv, abs/2103.10013, 2021. URL https://aple stt./pppi.smmttcsscchollaaa ort.s//ennnicccoooooooo///oooooo org/CorpusID: 232269939. (Cited on 4)
Alec Helbling, Mansi Phute, Matthew Hull, and Duen Horng Chau. LLM Self Defense: By Self Examination, L.Mss Know They Are Being Tricked. ArXiv, abs/2308.0777 073308 2023. URL attps://apps semanticcoani org/CorpuuID::2688744311 (Cited on 22)
Keegan Hines, Gary Lopez, Matthew Hall, Federico Zarfati, Yonatan Zunger, and Emre Kiciman. Defending Against Indirect Prompt Injection Attacks With Spotlighting. ArXiv, abs/2403.14720, 2024. URL
https://api.
semanticscholll httpattiiiiicoaaaooo/rrrrrrrooee orgg/CorpuSDD22666771
(Cited on 22)
Sanghyun Hong, Michael Davinroy, Yigitcan Kaya, Stuart Nevans Locke, Ian Rackow, Kevin Kulda, Dana Dachman-Soled, and Tudor Dumitras. Security Analysis of Deep Neural Networks Operating in the Presence Cache Side- Channel Attacks. ArXiv, abs/1810.03487,, 2018. URL https: //api. semanticscholar. / 52938514. (Cited on 17)
Zhang-War Wei Hong, Idan Shenfeld, Tsun-Hsuan Wang. Yung-Sung Chuang, Aldo Pareja, James Glass, Akash Srivastava, and Pulkit Agrawal. Curiosity-driven Red-Leaming for Large Language Models. 2024. URL /ttts:://pp..semmtiicsssee Balplaar.rrrr//ooo//re w 304. (Cited 28)
Chia-Yi Hsu, Yu-Lin Tsai, Chih-Hsun Lin, Pun-Yu Chen, Chia-Mu Yu, and Chun- Ying Huang. Safe lora: the silver lining of reducing safety risks when fine-tuning large language models. CoRR, /////2405..1888 2024. doi: 10. 40.48550/ 48550/ARXIV ARXIV 2445.16833. URL https://doi... org/10 48550/ 48550/arXiv. 2405. 16833. (Cited on 24)
Xing Hu, Ling Liang, Lei Deng, Shuangchen Li, Xinfeng Xie, Yu Ji, Yufei Ding, Chang Liu, Timothy Sherwood, and Yuan Xic. Neural network model extraction attacks in edge devices by hearing architoctural hints. ArXiv, abs/1903.03916, 2019. URL https://api. sew.ppissssaannncsseesaaeee wwweaaarrrrr/rooooaeeee org/CorpusID:73722 73729966. (Cited on 17)
Zhengmian Hu, Gang Wu, Saayan Mitra, Ruiyi Zhang, Tong Sun, Heng Huang, and Vishy Swaminathan. Token-Level Adversarial Prompt Detection Based on Perplexity Measures and Contextual Information. (Ciled on
ArXiu, Abs/2311.11500 2023. URL https://api. wwwantcsccollar.orrooo httpcicchhoaa.rrooorrrreeee wwwp//orpos.D:25200000 22)
Tiansheng Huang, Gautam Bhattacharya, Pratik Joshi, Josh Kimball, and Ling Liu. Antidote: Post-fine- tuning safety alignment for large language models against harmful fine-tuning. ArXiv, abs/2408.000 0.4000 2024a. URL https://api. semanticscholar. wwwphhholarrrro//orrooooeoooo /t:CCoouu::211102050000 (Cited on 24, 25)
Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Harmful fine-tuning attacks and defenses for large language models: A survey. CoRR, abs/2409.184 18169,, 2024b. doi: (0.48550/ARXI) 2409.18169. URL https: h/tp:///ooo.o/////
48550/arXiv 2409. 18169. (Cited on 20)
Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Booster: Tackling harmful fine-tuning for large language models via attemuating harmful perturbation. ArXio, abs/2409 abs/2409.01556, 2024c. URL https: //api. semanti otsp 272367199. (Cited on 24, 25)
Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Lazy safety aligmment for large language models against harmful fine-tuning. ArXiu, Abs/2200.18644 2024d. URL https: //api. semantischhollaar org. org/CorppsIID 270095345. (Cited on 24, 25)
Tianshong Huang, Sihao Hu, and Ling Liu. Vaccine: Porturbation-aware alignment for large language models against harmful fine-tuning attack. In The eiihth- Annual Conference on Neural Information Processing Systems, 2024e. (Cited 24, 25, 28)
43

---

Published in Transactions on Machine Learning Research (05/2025)
Tiansheng Huang, Sthao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Virus: Harmful fine-tuning attack for large language models bypassing guardrail moderation. 2025. URL https://api. http:/pp..semanicsscho.....aa org/CorppuSD::27555444, (Cited on 20)
Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation. Arxi abs/2310.06987, 2023a. URL https://api.
semanticscholar wttg/CorpusI::25355555.
(Cited on 4, 16, 23, 27)
Yue Huang, Qihui Zhang, Philip S. Yu, and Lichao Sun. TrustGPT: A Benchmark for Trustworthy and Responsible Large Language Models. ArXiv, abs/ abs/2330..1550 2023b. URL https://api. httpss//aaa.ssmmnnicschlll.aaa
org/CorpusID:259200) 259292452. (Cited on
Yujin Huang, Terry Yue Zhuo, Qiongkai Xu, Han Hu, Xingliang Yuan, and Chunyang Chen. Training-fres free Lexical Backdoor Attacks on Language Models. Proceedings of the ACM Web Conference 2023, 2023c.
URL https://apt: wttp:/..sematiiccchieee
(Cited on 4, 18)
Zhuoqun Huang, Neil G Marchant, Keane Lucas, Lujo Bauer, Olga Ohrimenko, and Benjamin Rubinstein. RS-Del: Edit distance robustness certificates for sequence classifiers via randomized deletion. Advances in Neural Information Processing Systems, 36, 2024f. (Cited on 23)
Evan Hubinger, Carson E. Denison, Jesse MU, Mike Lambert, Meg Tong, Monte Stuart MacDiarmid, Tamera Lanham, Daniel M. Ziegler, Tim Maxwell, Newton Cheng, Adam Jermyn, Amanda Askell, Ansh Radhakrishnan, Cem Anil, David Kristjanson Duvenaud, Deep Ganguli, Fazl Barez, Jack Clark, Kamal Ndousse, Kshitij Sachan, Michael Sellitto, Mrinank Sharma, Nova DasSarma, Roger Grosse, Shauna Kravec, Yuntao Bai, Zachary Witten, Marina Favaro, Jan Markus Brauner, Holden Karnofsky, Paul Francis Christiano, Samuel R. Bowman, Logan Graham, Jared Kaplan, Sören Mindermann, Ryan Greenblatt, Buck Shlegeris, Nicholas Schiefer, and Ethan Perez. Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training. ArXie, abs/2401 105566, 2024. URL https://app semanticscholar. org/ CorpusID: 266933030. (Cited on 24, 29)
Chia-Chien Hung, Wiem Ben Rim, Lindsay Frost, Lars Bruckner, and Carolin Lawrence. Walking a Tightrope Evaluating Large Language Models in High-Risk Domains. ArXiv, abs/2311. 2023. URL https: //api. semanticscholar htrg/CorpusI:: 265456842. (Cited on 29)
Hakan Inan, K. Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama Guard: LLM-based Input-Output Safeguard for Human-AT Conversations. ArXie, abs/2312.00 CorpusID: 2661 74345. (Cited on 3, 22, 27)
2023. URL https: //api. semanticscholarr org/
Nanna Inie, Jonathan Stray, and Leon Derezynski. Summon a Demon and Bind it: A Grounded Theory of LLM Red Teaming in the Wild. ArXio, abs/2311.06237,, 2023. URL https://api. semanticscholar. org/ (Cited on 5, 6)
CorpusID: 265112805..
JailbreakChat. JailbreakChat, http:://wwwwss.ooooooooccc////aa
2023. (Cited on 10)
Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping ych Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Baseline Defenses for Advensarial Attacks Against Aligned Language Models. ArXiv, org/CorpusID: 261 494182. (Cited on 22)
2023a. URL https://apt sennaticcnemt
Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert P. Dick, Hidenori Tanaka, Edward Grefenstette, Tim Rocktaschel, and David Krueger. Mechanistically analyzing the effects of fine-tuning procedurally defined lasks. ArXio, abs/2311 1.11766,, 2023b. URL https://api. /ann:://pp..mmmattiii
org www./ooppsso......... :ttrpussD::25888855 265308865.
(Cited on 20)
Eric Jang, Shixiang Shane Gu, and Ben Poole. Categorical Reparamederiiation with Gumbel-Softtax. ArXiu, abs/1611.00 111111111001111 2016. URL http://pp...mmanncccccolaar..rr. hwt.:/arr.rrrgrrruuuD/22222///1 (Cited on 21)
Jessica Ji. What does AT Red- Tea--Te mean?). 2023. (Cited on 2)
44

---

Published in Transactions on Machine Learning Research (05/2025)
Jiabao Ji, Bairu Hou, Alexander Robey, George Pappas, Hamed Hassani, Yang Zhang, Eric Wong, and Shiyu Chang. Defending Large Language Models against Jailbreak Attacks via Semantic Smoothing. ArXin, abs/2442.16192, 2024. URL https://app..mmmniccchooooo
semanticscoolar
org/CorpusID:
267938332. (Cited on 22, 27)
Bojian Jiang, Yi Jing, Tianhao Shen, Qing Yang, and Deyi Xiong. DART: Deep Adversarial Automated Red Teaming for LLM Safety. 2024. URL http://pp...mmmticccccooo.rrrrr. wrwp/oorpsss:: 2710391133 (Cited on 26)
Haibo Jin, Ruoxi Chen, Andy Zhou, Jinyin Chen, Yang Zhang, and Haohan Wang. GUARD: Role-playing to Generate Natural-language Jailbreakings to Test Guideline Adherence of Large Language Models. ArXin, abs/2442.03229, 2024. URL https://apl: https://api.semmniicctoie swwanticscholaar.. httprn/ssccoooaooor//rrrpssssses org/CorpusII: 267411743. (Cited on 4)
Zhi Jing, Yongye Su, Yikun Han, Bo Yuan, Haiyun Xu, Chunjiang Liu, Kehai Chen, and Min Zhang, When Large Language Models Meet Vector Databases: A Survey. ArXiu, abs/2402.017633 2024. URL
267412060 (Cited on 16)
JointTaskForce. Security and privacy controls for information systems and organizations. Technical report, National Institute of Standards and Technology, 2017. (Cited on 31)
Erik Jones, Anca D. Dragan, Aditi Raghunathan, and Jacob Steinhardt. Automatically Auditing Large Language Models via Discrete Optimization. ArXiv, abs/2303.04381, 2023. URL https: seatiisemat org/CCrppsID::20 CorpusID:257405443 257405439. (Cited on 21)
G Joy Persial, M Prabhu, and R Shanmugalakshmi. Side channel attack-survey. Int. J. Adv. Sei. Res. Rev, (4):54-57, 57, 2011. (Cited on 16)
Alkis Kalavasis, Amin Karbasi, Argyris Oikonomou, Katerina Sotiraki, Grigoris Velegkas, and Manolis Zampetakis. Injecting Undetectable Backdoors in Deep Learning and Language Models. 2024. URL
https://pp...sattttccccaarrrrrrr 270370992. (Cited on 19)
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating Training Data Mitigates Privacy Risks in Language Models. ArXiv, abs/2202.06539, 2022. URL https://api. wwwpaaatiiiccoooaroorr/rrrereeee 246823128. (Cited
16, 25)
Nikhil Kandpal, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Backdoor Attacks for In-Context Learning with Language Models. ArXiv, abs/2307.14692, 2023. URL http:s/////pi.ssmmaiicccch....l org/ CorpusID: 25000030777 (Cited on 19)
Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei A. Zaharia, and Tatsunori Hashimoto. Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks. ArXiv, abs/2302.057333, 2023. URL https://app. sew.pppisseaniicchohcllaaa http::iiiccccolaoooorrrrrrrrreee wwtp//orpss//oooo...oooo 256827723.. (Cited on 4, 12)
Mintong Kang, Nezihe Merve Gürel, Ning Yu, Dawn Song, and Bo Li. C-rag: certified generation risks for retrievallaugmmented language models. In Proceedings of the 41st International Conference on Machine Learning, ICML/24. JMLR.org, 2024a. (Cited on 23, 28)
Mintong Kang, Nezihe Merve Gürel, Ning Yu, Dawn Song, and Bo Li. C-RAG: certified generation risks for relrievalanugmented language models. In Forby-first International Conference on Machvine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024b. URL id-FMa4c5NhOe. (Cited on 18)
Andrej Karpathy, Intro to Large Language Models. on 17, 18)
2023. (Cited
Roose Kevin. Bing's A.I. Chat: 'I Want to Be Alive. thie-chabbtt-rrnsccce html, 2023a. (Cited on 2)
Roose Kevin. A Conversation With Bing's Chatbot Left Me Deeply Unsettled. 202//8//0/ccchooogg///nggcaaaaae ottps////nn.iiccc..oooooorotttto html, 2023b. (Cited on 2)
45

---

Published in Transactions on Machine Learning Research (05/2025)
Jinhwa Kim, Ali Derakhshan, and Ian G. Harris. Robust Safety Classifier for Large Language Models: Adversarial Prompt Shield. ArXiu, 0bs/2311.00172 2023. URL nttps://apl: semanticscholar.orrg
CorpusID:264833136. (Cited on 27)
Jason Koebler. Asking ChatGPT Repeat Words 'Forever' Is Now a Terms of Service Viola- tion.
2023. (Cited 14)
Andreas Kopf, Yanmic Kilcher, Dimitri von Rutte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Rich ard Nagyfi, ES Shahul, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. OpenAssistant Conversations Democratizing Large Language Model Alignment. ArXiv, 0bs/2304.07337,, 2023. (Cited on 6)
Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L. Buckley, Jason Phang, Sam Bowman, and Ethan Perez. Pretraining Language Models with Human Preferences. ArXiv, Abs/2302.08582, 2023. URL https: / httt:::://pi.smmatiicssss sewanticscholar http:ooooo///g///0000. 2570000055. (Cited on 3)
George Kour, Marcel Zalmanovici, Naana Zwerdling, Esther Goldbraich, Ora. Nova Fandina, Ateret Anaby- Tavor, Orna Raz, and Eitan Farchi. Unveiling Safety Vulnerabilities of Large Language Models. ArXiv, abs/ abs/2311.04124, 2023. URL https://appl.eemnniicchholl..
: www.sti .rrg :orppsID:: 265042968. (Cited on 26)
Satyapriya Krishna, Chirag Agarwal, and Himabindu Lakkaraju. Understanding the Effects of Iterative Prompting on Truthfulness. CoRR, abs 0////222200660 2024. doi: 10.48550/ARAN 2402.06625. URL
https://6oi.orr//0 hetp///1..8550/// /arivv. 2402. 06625. (Cited on 2)
Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Soheil Feizi, and Hima Lakkaraju. Certifying LLM Safety against Adversarial Prompting. CoRR, abs/2309.022 02000 2023. doi: (Cited on 23, 28)
URL
https://doi.orrg/1 08880485550//.0 2309. 2309.02705.
Kcita Kurita, Paul Michel, and Graham Neubig. Weight Poisoning Attacks on Pretrained Models. ArXio, wwt.eeanttiicccool.rrrrrro/roroo www.orrrr/oSs:///111 215755332.. (Cited on 19)
abs/2004.06660, 2020. URL
Sander Land and Max Bartolo. Fishing for Magikarp: Antomatically Detecting Under-trained Tokens in Large Langnage Models. 2024. URL https://api. semaaticcccnn wwtp:/Crrrso/...22655577777 (Cited on 12)
Raz Lapid, Ron Langberg, and Moshe Sipper. Open Sesame! Universal Black Box Jailbreaking of Large Language Models. ArXie, 0bs/2309.01446, 2023. URL https://api. wwwpaantiiiccolaa.r///oo/rooroue 261530019. (Cited on 4, 13)
Deokjae Lee, Jeone Lee, Jung- Woo Ha, Jin-Hwa Kim, Sang- Woo Lee, Hwaran Lee, and Hyun Oh Song. Query-Efficient Black Box Red Teaming via Bayesian Optimization. ArXiv, abs/2305.114444, 2023. URL wwwantiescoolllaat...eeee wowg/CorpuusD/i 25896043.. (Cited on 4, 12)
https::/api..emmnticcchhh.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Cillionn-Burch, and Nicholas Carlini. Deduplicating Training Data Makes Language Models Better. In Annual Meeting of the Association for Computational Linguistics, 2021. URL https::/ //api. semanticscholar. 235829052. (Cited on 16, 25)
httpttisccccooooooooooooooooeeee coapllaroorg/oppuuuuueoe org/CCopusID:
Alyssa Lees, Vinh Q. Tran, Yi Tay, Jeffrey Scott Sorensen, Jai Gupta, Donald Metzler, and Lucy Vasserman. A New Generation of Perspective API: Efficient Multilingual Character-level Transformers. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022. URL https:
segpiiismmanicschollee hammaaniicccclla...ooo//oorppppo orgg/oorpsID: (DMrpusID:247058801 247058801. (Cited on 22, 27)
Jan Leike, David Krucger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: research direction. CoRR, abs/1811.07771, 2018. URL http: org/abs/1811 07871. (Cited on 23)
46

---

Published in Transactions on Machine Learning Research (05/2025)
Benjamin Lemkin. Using Hallucinations to Bypass GPT4's Filter. 2024. URL http:::/app...mmaaticchhoooo
PBB/CorpuSDDD:28358777 (Cited on 4)
Simon Lermen, Charlie Rogens-Smith, and Jeffrey Ladish. LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B. ArXiu, abs/2310.20624, 2023. URL https///ppp..eemattsscccolarrrra 264808400. (Cited 20)
Cheng Li, Jindong Wang, Kaijie Zhu, Yixuan Zhang, Wenxin Hou, Jianxun Lian, and Xingxu Xie. Large Language Models Understand and Can be Enhanced by Emotional Stimuli. 2023a. URL https://api. semanticscholar. org/CorpusID: 2601 26019. (Cited on 11)
Haoran Li, Mingshi Xu, and Yangqiu Song. Sentence embedding leaks more information than you expect: Generative embedding inversion attack to recover the whole sentence. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Findings of the Association for Compulational Linguistics: ACL 2023, pp. 14022 14040, Toronto, Canada, July 2023b. Association for Computational Linguistics. doi: 10.18653/v1 4, 16)
org/2023. (w12222...innnng findings-acl. 881. (Cited on ac1.881. URL https: 2twpaatooooooooooo/22233333
Lijum Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. SALAD-Benne BenchB A Hicrarchical and Comprehensive Safety Benchmark for Large Language Models. ArXie, abs/2402.050444, 2024a. URL https: semanticsctin ciccolllarr hwtphholaarrggg/orreooree arg/CorpusID::87522455 (Cited on 29)
Linyang Li, Demin Song, Xiaonan Li, Jichang Zeng, Ruotian Ma, and Xipeng Qiu. Backdoor Attacks Pre-trained Models by Layerwise Weight Poisoning. In Conference Empirical Methods in Natural Language Processing, 2021a. URL https://apt http:///...aatt.t.toooooooooooo/ 19)
230orruusID:2233655555 (Cited on
Nathaniel Li, Ziwen Han, lan Steneker, Willow Primack, Riley Goodside, Hugh Zhang, Zifan Wang, Cristina Menghini, and Summer Yuc. LLM defenses are not robust to multi-turn human jailbroaks yet. CoRR, abs/2408. 15221, 2024b. doi: 15221. (Cited on 26, 31)
URL https://doi. org/10. 48550/arXiv http//axivvi....... 2408.
Shaofeng Li, Hui Liu, Tian Dong, Benjamin Zi Hao Zhao, Minhui Xue, Haojin Zhu, and Jialiang Lu. Hidden Backdoors in Human-Centric Language Models. Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, 2021b. URL https://apt semanticscholar ormamnnicccchlaaaoooo/r/rrrssss 233481877. (Cited on 19)
Tianlong Li, Xiaoqing Zheng, and Xuanjing Huang. Open the Pandora's Box of LLMs: Jailbreaking LLMs through Representation Engineering. ArXív, abs/ 2bs//400106824,, 2024c. URL https: //api. semanticscholar. org/CorpusID: 266999568. (Cited on 4)
Xiaoxia Li, Siyuan Liang, Jiyi Zhang, Hansheng Fang, Aishan Liu, and Ee-Chien Chang. Semantic Mirror Jailbreak: Genetic Algorithm Based Jailbreak Prompts Against Open-source LLMs. ArXie, 2b872402.14872, 2024d. URL /taps: http:://pap..saaniieeeeee semanticscholar
semmaniiiscchol.er...r///rrouuop wttp://ooo/ooo./////// 2678973711 (Cited on 13)
Xucchen Li, Florian Tramèr, Percy Liang, and Tatsunori B. Hashimoto. Large Language Models Can Be Strong Differentially Private Learners. Arviv, abs/2110.05669, 2021c. URL /ttp:::///a...mmttiiiiii
org/CorpusID: 238634219, (Cited on 25)
Yanzhou Li, Shangqing Liu, Kangjie Chen, Xiaofei Xie, Tianwei Zhang, and Yang Liu. Multi-largel Backdoor Attacks for Code Pre-trained Models. In Annual Meeting of the Association for Computational Linguistics, 2023c. URL https: /ttts::/apptt //api. www.nticscholttoooo www.choaaa....r/oooottttt 259I:ID:::5999955 2591651344
(Cited on 19)
Yiming Li, Baoyuan Wu, Yong Jiang, Zhifeng Li, and Shutao Xia. Backdoor Learning: Survey, IEEE Trons-
actions on Neural Networks and Learning Systems, 35:5 22, 2020. URL https: https::/api..smmntttee seantiiscaat.
org/CorpusID: 22863311. (Cited on 19)
Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Ta Xia. Backdoor learning: A survey, IEEE Transactions on Neural Networks and Learning Systems, 2022. (Cited 19)
47

---

Published in Transactions on Machine Learning Research (05/2025)
Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. RAIN: Your Language Models Can Align Themselves without unnng ArXiv, abs/2309.07122,, 2023d. URL httpss//appt http:/pp...smaniisscholo.olrr org/LorpusID::21705555. (Cited on 24, 26)
Zongjie Li, Chaozheng Wang, Shuai Wang, and Cuiyun Gao. Protecting Intellectual Property of Large Language Model-Basse Based Code Generation APIs via Watermarks. Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, 2023e. URL
CorpusID: 265352433. (Cited on 14)
Jiawei Liang, Siyuan Liang, Man Luo, Aishan Liu, Dongchen Han, Ee-Chien Chang, and Xiaochun Cao. VI-Trooja:: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models. Arxiv, abs/2402.13885 2024. URL https://apl semanticscholar, wtw.iiicccoooooo......ooooo org/CorpusID:22778833. (Cited on 6)
Zeyi Liao and Huan Sun. AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs. 2024. URL sem.//pcpi.smmniiccccooolaaa org/ CorpuSID:269043100 (Cited on 4)
Lizhi Lin, Honglin Mu, Zenan Zhai, Minghan Wang, Yuxia Wang, Renxi Wang, Junjie Gao, Yixuan Zhang, Wanxiang Che, Timothy Baldwin, Xudong Han, and Haonan Li. Against The Achilles' Heel: A Survey on Red Teaming for Generative Models. 2024. URL https: / semanticscholaat ortp/CrrpsSI: 2688200098.8 (Cited on 6)
Aiwei Liu, Leyi Pan, Yijian Lu, Jingjing Li, Xuming Hu, Lijie Wen, Irwin King, and Philip S. Yu. A survey of text watermarking in the era of large language models. ACM Comput. Surv., 57:47:1-47:36, 2023a. URL https://api..emantiicchh.la www.stccscnnollla.t. Bac.lla..rrr//ppppeee org/CorpusID::200 2661 26616666111 91530. (Cited on 6, 14)
Guozhi Liu, Weiwei Lin, Tiansheng Huang, Ruichao Mo, Qi Mu, and Li Shen. Targeted vaccine: Safety alignment for large language models against harmful fine-tuning via layer-wise perturbation. ArXie, abs/2 abs/2440.09960, 2024a. URL https://api. semanti wwwhanaaascco.....ccoomoom wttptaaa...rrrg//orrssSSSS (CCorppsID:23334222400 (Cited on 24, 25, 28)
Jiaxu Liu, Xiangyu Yin, Sihao Wu, Jianhong Wang, Meng Fang, Xinping Yi, and Xiaowei Huang, Tiny Refinements Elicit Resilience: Toward Efficient Prefix-Model Against LLM Red- ood-Teaming 2024b. URL 26992999.. (Cited on 27)
Qin Liu, Fei Wang, Chaowei Xiao, and Muhao Chen. From Shortcuts to Triggers: Backdoor Defense with Denoised PoE. ArXiv, 149110 2023b. URL https::/aap..ssmenticcchoo..... httpcccchhooaaooooooosueseee org/CorpusID: 258866191. (Cited on 28)
semanticscholaat
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. ArXiv, abs/2309.06633 2023c. URL https::/api..... semanticchcm. wwwpsscholaa..oo/oooouuuuuo orgg/CrppusI: 261705578. (Cited on 23)
Tong Liu, Yingjie Zhang, Zhe Zhao, Yinpeng Dong, Guozhu Meng, and Kai Chen. Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstruction. ArXio, abs/2442.18104, 2024c. URL https://api. wwwanticscholaat htt.a//eeeeeattcoooooooooooooore
268041845. (Cited on 4)
Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models. CoRR, abs/ hbs/2310004451,, 2023d. doi: (0.48550/ARM8) 2310.04441. URL https://doi. org/10. wwwg///..550//rree/e 48550/arXiv.
2310. 04451. (Cited on 4, 10, 13, 14)
Xiaogeng Liu, Peiran Li, Edward Suh, Yevgeniy Vorobeychik, Zhuoqing Mao, Somesh Jha, Patrick McDaniel, Huan Sun, Bo Li, and Chaowei Xiao. Autodan-turbo: A lifelong agent for strategy self-exploration jailbreak 1ms. CoRR, abs/ 2410.05299,, 2024d. doi: 10.48550/ARXIV, 24100.0552.5. URL https: / //toi.or///////o// 10. 48550/arXiv 2410 05295. (Cited on 13)
48

---

Published in Transactions on Machine Learning Research (05/2025)
Xiaogeng Liu, Zhiyuan Yu, Yizhe Zhang, Ning Zhang, and Chaowei Xiao. Automatic and Universal Prompt Injection Autacks against Large Language Models. 2024e. URL https://api..... https///pp..emmatticscoo.... CorpusID:268296113. (Cited on 4)
Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, and Yang Liu. Prompt Injection attack against LLM-integrated Applications. CoRR, abs/ 22//2/30000999 2023e. doi: 10.48550/
ARXIV 0...8550/ARXII
URL https: //tp:.//oiioooo org/10 00..8805//ari1 2306. 05499. (Cited on 4, 13)
Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. Jailbreaking ChatGPT via Prompt Engincering: An Empirical Study. ArXiv, abs/2305.13860, 2023f. URL attps://apis sewanticscholaatee htt.:/ooa...rggg/oooooosD:/:://
(Cited on 4, 10)
Yingqi Liu, Guangyu Shen, Guanhong Tao, Shengwei An, Shiqing Ma, and X. Zhang. Piccolo: Exposing Complex Backdoors in NLP Transformer Models. 2022 IEEE Symposium Security and Privacy (SP), pp. 2025-2042, 2022. URL https://apt semanticscholar oatpaaniicccooooooooo//ooooooD: 248067917. (Cited on 19)
Zichuan Liu, Zefan Wang, Linjie Xu, Jinyu Wang, Lei Song, Tianchun Wang, Chunlin Chen, Wei Cheng, and Jiang Bian. Protecting Your LLMs with Information Bottleneck. ArXiv, abs/2404.13968, 2024f. URL
2692935911. (Cited on 27)
LLM Agents. LLM Agents. https://www.ppomnnigguudaaaaaa htw.:///wwwa.eeeeeee///aaaaaaaaa lmm-ggents, 2023. (Cited on 7, 21)
LLM Security. LLM Security, https://llnssecurty.neet/ 2023. (Cited on 22)
Renze Lou, Kai Zhang, and Wenpeng Yin. A Comprehensive Survey on Instruction Following, 2023. URL https://app http::/p....aaaaiiccooooooooro/r 257632435. (Cited on 3)
Dong Lu, Tianyu Pang, Chao Du, Qian Liu, Xianjun Yang, and Min Lin. Test-Time Backdoor Attacks on Multimodal Large Language Models. 2024a. URL http:://api..smatticchhoo.rr httpatttiiccooaaoog//rrrrrssssss org/CorpusID: 267637232. (Cited on 4, 18)
semantccscoola
Lin Lu, Hai Yan, Zenghui Yuan, Jiawen Shi, Wenqi Wei, Pin-Yu Chen, and Pan Zhou. AutoJailbreak: Exploring Jailbreak Attacks and Defenses through Dependency Lens. 2024b. URL https://api. semanticscholar o
(Cited on 25)
Huijic Lv, Xiao Wang, Yuan Zhang, Caishuang Huang, Shihan Dou, Junjic Ye, Tao Gui, Qi Zhang, and Xuanjing Huang. CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Models. ArXiv, abs/2402.16717,, 2024. URL
org. orggCorrpsID: 2680333344,
(Cited on 4, 13)
Chengdong Ma, Ziran Yang, Minquan Gao, Hai Ci, Jun Gao, Xuehai Pan, and Yaodong Yang. Red Teaming Framework for Red Teaming Language Models. ArXiv,
Game: A
2023.
URL //app: semanticscholar. wor.:laarrrrggoooousIi.... 263334034. (Cited on 26)
Chris Maddison, Andriy Mnih, and Yee Whye Teh. The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables. ArXiv, abs/1611.00712, 2016. URL https://apt semanticscholar org/
CorpusID: 14307651. (Cited on 21)
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks. ArXiv, Abs/1706.06083, 2017. URL https:
/t 3488815. (Cited on 24)
Vahid Majdinasab, Michael Joshua Bishop, Shawn Rasheed, Arghavan Moradi Dakhel, Amjed Tahir, and Foutse Khomh. Assessing the Security of GitHub Copilot Generated Code A Targeted Replication Study. ArXiu, abs/231 bbs/2211.11177, 2023. URL https::/app swwanticscholar.c
wwtpolaaarrrg/orrrggeee http/CoopuII::25555555. https:::/appisennnniccchoorr
(Cited on
2)
Neal Mangaokar, Ashish Hooda, Jihye Choi, Shreyas Chandrashekaran, Kassem Fawaz, Somesh Jha, and Atul Prakash. PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Rails. ArXiv, abs/2402.1599 15911, 2024. URL https: //api. http:/cccooo....oo///oooomaaaaaa (Cited on
21)
49

---

Published in Transactions on Machine Learning Research (05/2025)
Todor Markov, Chong Zhang, Sandhini Agarwal, Florentine Eloundou Nekoul, Theodore Lee, Steven Adler, Angela Jiang, and Lilian Weng. A Holistic Approach to Undesired Content Detection in the Real World. Proceedings of the AAAI Conference on Artificial Intelligence, 37(12):15009 15000000--111,,, Jun. 2023. doi: 16699/aai..v. haai.v37:12.267555 URL https://ojs aaai www.lorgggieeeeeeee pttp 22)
attpaa//ww..........aaaaaaaataaa http:/AAAAAarrtccl////ee/////// (Cited on
Thomas Marshall, Adam Scherlis, and Nora Belrose. Refusal in 1lms is an affine function. CoRR, abs/2 0tss/2411.09003 2024. doi: 10.48550/ARM5 /tt.//RRXII...1..10.1 241..9903.. URL https://doi. org/10. 10.5000////./iii 2411. 09003. (Cited on 18)
Justus Mattern, Fatemehsadat, Mireshghallah, Zhijing Jin, Bernhard Schoelkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrick. Membership inference attacks against language models via neighbourhood comparison. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguisties: ACT. 2023, pp- 11330-11343, Toronto, Canada, July 2023. Association .
for Computational Linguistics. doi: 201553/1122223/2/ hin./iiinnn..nnng--oiiaaao acl..19 URL
3023.findings-ae 719. (Cited on 4)
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. ArXie, abs/2402.04249, 2024. URL
www..ogg//oooo 2Ttp/CCopuSDD::2449979.. 267499790. (Cited on 3, 29)
Matthicu Meeus, Shubham Jain, Marek Rei, and Yves-Alexandre de Montjoye. Did the neurons read your book? document-level membership inference for large language models. arXiv preprint arXiv:2310.1500 2023. (Cited on 15)
Ninareh Mehrabi, Palash Goyal, Christophe Dupuy, Qian Hu, Shalini Ghosh, Richard S. Zemel, Kai-Wei Chang, Aram Galstyan, and Rahul Gupta. FLIRT: Feedback Loop In-context Red Teaming. CoRR, abs/ / 2338.04265 2023a. doi: 10.48550/ARX8 2306.04265. URL https: //doi. http::///ooooo//...//// 0rg/10.48550/ 48550/arXiv. arxiv. 2308. 04265. (Cited on 4, 12)
Ninarch Mchrabi, Palash Goyal, Anil Ramakrishna, J. Dhamala, Shalini Ghosh, Richard Zemel, Kai-Wei Chang, A. G. Galstyan, and Rahul Gupta. JAB: Joint Adversarial Prompting and Belief Augmentation. ArXiv, abs/2311.09473, 2023b. URL https://api. semanticscholar org/CorppusD::26522637 (Cited on 26)
Anay Mchrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of Attacks: Jailbreaking Black- Box LLMs Automatically, ArXiv, abs/2312.021199, 2023. URL
(Cited on 4)
Kai Mei, Zelong Li, Shuyuan Xu, Ruosong Ye, Yingqiang Ge, and Yongfeng Zhang. LLM Agent Operating System. 2024a. URL https://api. semanticscholar. sonmatttiiccoollr.oor//roroooooo 268681499. (Cited on 21)
Lingrui Mei, Shenghua Liu, Yiwei Wang, Baolong Bi, Jiayi Mao, and Xueqi Cheng. 'Not Aligned'is Not' Malicious': Being Careful about Hallucinations of Large Language Models' Jailbreak. 2024b. URL hittps://apt http:/ppp...smmanicsccccco...aae wawpaaarrrg///oosttt22000000
(Cited on 29, 31)
Peter Mell, Karen Scarfone, and Sasha Romanosky, A complete guide to the common vulnerability scoring system version 2.0, 2007-07-30 2007. URL https:// http://sppp.sss..ooo//////////// httpst//oo//bbbbiitiiiooege.edde 51198. (Cited on 30)
wttp://teepp.ccccpu..ooooo ctt.:ubb_pppcccccc idd=
Meta. Overview of Meta Al safety policies prepared for the UK Al Safety Summit. https:/ wwc.a//eengg//pllcciisiaaaaaaaaa fortsssss
2023. (Cited on
Microsoft. The Python Risk Identification Tool for generative AI (PyRIT). httpss//giiaub..om///ne///////tt 2023. (Cited 29)
50

---

Maximilian Mozes, Xuanli He, Bennetl Kleinberg, and Lewis D. Griffin. Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilities. ArXiv, abs/2308. 1283333388883,, 2023. URL https://api: htt.:///r....rr//roorpo/::111110 Published in Transactions on Machine Learning Research (05/2025)
Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor
and Reza Shokri. Quantifying privacy risks of masked language models using membership inference attacks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 8332 8347, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https: //aclanthology. org/2022. wttp://02..mmmmmmmmm....... 570. (Cited on 15)
Lingbo Mo, Zeyi Liao, Boyuan Zheng, Yu Su, Chaowei Xiao, and Huan Sun. A Trembling House of Cards? Mapping Adversarial Attacks against Language Agents. ArXie, abs/2402.10196, 2024a. URL
267682286. (Cited on 21)
Yichuan Mo, Yuji Wang, Zeming Wei, and Yisen Wang. Studious Bob Fight Back Against Jailbreaking via Prompt Adversarial Tuning. ArXiv, abs/2402.06255, 2024b. URL https://api. seaaaesssatttttcccoorooooroo
CorpusID:266617133 (Cited on 22, 27)
Model Denial of Service. Model Denial of Service. modeel-denial-ofseriieee 2023. (Cited on 9)
John Morris, Volodymyr Kuleshov, Vitaly Shmatikov, and Alexander Rush. Text Emboddings Reveal (Almost) As Much As Text. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12448-12444 Singapore, December 2023a. Association for Computational Linguistics. doi: 10.18653 http//222.eemlp-peeeee https://aclaanthlogr. org/202g emnlp-mmain.. 765. (Cited on 15, 16, 25)
mmaip....ml.mmiin............. main. 765. URL
John X. Morris, Volodymyr Kuleshov, Vitaly Shmatikov, and Alexander M. Rush. Text embeddings reveal (almost) as much as text. In Conference on Empirical Methods in Natural Language Processing, 2023b. (Cited on
URL https://apis http:ppp.ssmmaiisschhlll.aa .rrg httg/CoopuSD::232290000
John X. Morris, Wenting Zhao, Justin T. Chiu, Vitaly Shmatikov, and Alexander M. Rush. Language Model Inversion. ColR,, abs /t////311..644411 2023c. doi: 10.48550/ARXIV22111
URL
48550/arXiv. 2311. 13647. (Cited on 4, 15)
(Cited on 22)
Arif Ali Mughal. The art of cybersecurity: Defense in depth strategy for robust protection. In- lernational Journal of Intelligent Automalion and Computing, 1(1):1 20, Mar. 2018. URL https: //research. wtw.ercccttennnggtee.oo. httprr//iiwi.e.oo//////////taaaa
(Cited on 30, 31)
Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhao- han Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel Jaymin Mankowitz, Doina Precup, and Bilal Piot. Nash learning from human feedback. ArXiv, abs/ 22112.00886 2023. URL https://app semanticscholast wwtptttiicccoooaaorrrrorrarreeee org/CorpusID: 265629682. (Cited on 23)
Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable Extraction of Training Data from (Production) Language Models. ArXiv, abs/2311.17032, 2023. URL https:
//api.
(Cited on 4, 14)
NIST CVE. NIST CVE. https://nvd.aist.oov//// 2022. (Cited on 3)
Zhenxing Niu, Haodong Ren, Xinbo Gao, Gang Hua, and Rong Jin. Jailbreaking Attack against Multimodal Large Language Model. ArXiv, abs/2402.02309. 2024. URL https://ap) semanticcchclll... ott 267413270. (Cited on 6)
org/CorpusID:
Munachiso Nwadike, Takumi Miyawaki, Esha Sarkar, Michail Maniatakos, and Farah Shamout, Explainability matters: Backdoor attacks medical imaging. Xiu preprint arXio:2101.00008 2020. (Cited 5)
51

---

Published in Transactions on Machine Learning Research (05/2025)
Myung Gyo Oh, Leo Hyun Park, Jacuk Kim, Jaewoo Park, and Tackyoung Kwon. Membership Inference Attacks With Token-Level Deduplication on Korean Language Models. IEEE Access, 11:10207-10211 2023. (Cited on 15)
Seong Joon Oh, Maximilian Augustin, Mario Fritz, and Bernt Schiele. Towards Reverse- Reverse-Ungineering Black-Box Neural Networks. In International Conference on Learning Representations, 2017. URL
wttps/hhoaar.orr//rrrsssaa222//2 (Cited on 14)
OpenAl. Introducing ChatGPT. https://openaiicom, http.//////ooo/ccaht........ 2022. (Cited 6, 13)
OpenAL. GPT-4 Technical Report. 2023. http://s///g....ttttttaaaaorooro 257.52228.....
(Cited on 1)
OpenAl. Function Calling OpenAl. http://///ooooooooooo//////////r 2023. (Cited
on 12)
OpenAT. Using logit bias define loken probability. Bogit-bias-Be www.--battt-d--f--tooeeeee potepi--ttkee--rrbbbllltt 2023. (Cited on 15)
OpenAl. OpenAl Red Teaming Network.
2023. (Cited on 30)
OpenAL. GPT4 Fine Tuning.
2024a. (Cited on 20)
OpenAL. OpenAI Moderation Endpoint. ppt.dmm...aaaaaccco///////////im 2024b.
(Cited on 3, 22, 27)
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training Language Models to Follow Instructions with Human Feedback. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.); Advances Neural Information Processing Systems, 2022a. URL (Cited on 3)
https://openreview. wttp://rrriiieeeeeeeeruuuuuuae
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Newral Information Processing Systems, 15:27730-277444 2022b. (Cited on 3, 23)
OWASP. 2025 Top 10 Risk Mitigations for LLMs and Gen Al Apps. top-10/, 2025. (Cited on 3)
Mustafa Safa Ozdayi, Charith S. Peris, Jack G. M. FitzGerald, Christophe Dupuy, Jimit Majmudar, Haidar Khan, Rahil Parikh, and Rahul Gupta. Controlling the Extraction of Memorized Data from Large Language Models via Crompt-Tunnng: In Annual Meeting of the Association for Computational Linguistics, 2023. htcC//ooosii:ee::::00
URL https: //ppi. semant i cicschllar.
(Cited on 4, 25)
Nicolas Papernot, Patrick Medaniel, Ian J. Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. Practical Black-Box Attacks against Machine Learning. Proceedings of the 2017 ACM on Asia Conference on Computer and Commmnnications Security, 2016. URL https://api. semanti semanticscholar. Org/CorpSSID: 1090603. (Cited on 14)
Nicolas Papernot, Patrick Medaniel, Arunesh Sinha, and Michael P. Wellman. SoK: Security and Privacy in Machine Learning. 2018 IEEE European Symposium on Security and Privacy (EuroSEP), pp. 399-414 414, 2018. URL
(Cited on 9, 10, 11)
Dario Pasquini, Martin Strohmeier, and Carmela Troncoso. Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attacks. ArXiv, abs/2408.03792, 2024. URL https://api.
smmatticschlaar..og//oooooo hrtp.///rrppssS02222253244 (Citee on 4)
52

---

Published in Transactions on Machine Learning Research (05/2025)
Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large Language Model Connected with Massive APIs. ArXiv, abs/2305. 15334, 2023. URL bttpss///ap: www.piisemanicccoolaa...oo org/ CorpusID:258865188 84. (Cited on 7)
Sonali Pattnaik, Rohan Karan, Srijan Kumar, and Clémentine Fourrier. Introducing the Chatbot Guardrails Arena. httpss//huuginngfacec http://c....oo/////eeee/e weagg//rnn--iighhhen 2024. (Cited on 29)
Anselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos, and Yuandong Tian. AdyPrompter:
Fast. Adaptive Adversarial Prompting for LLMs. ArXiv, abs/2404.16873,, 2024. URL https://api.
semanticscholar. org/CorpusII:
oemmattissccoler..rr/////ooooooo wtt.:///orruss..:254330999999999 269430799. (Cited on 14)
Will Pearce and Joseph Lucas. NVIDIA AI Red Team: An Introduction. https://developer.nviiii
blog/nvidia-ai- aetedd-tea-aa-ittrtoe
an
2023. (Cited on 2)
Kellin Pelrine, Mohammad Taufeeque, Michal Zajkac, Euan McLean, and Adam Gleave. Exploiting Novel GPT-4 APIs. 2023. URL https://ap...mmattiiioe. http:://g...saaooatsssrrrrrrrrrp 205. (Cited on 4, 12)
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra- Aimée Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only. ArXiv, /bs/2306001116,, 2 (Cited on 16)
2023. URL
Sheng-Hsuan Peng, Pin-Yu Chen, Matthew Hull, and Duen Horng Chau. Navigaling the Safely Landscape: Measuring Risks in Finetuning Large Language Models. ArXie, abs/2405.17374, 2024a. URL https: / segan
wrwg/CorpussD.... 270063157. (Cited on 20)
Shengyun Peng, Pin-Yu Yu Chen, Matthew Hull, and Duen Horng Chau. Navigating the safety landscape: Measuring risks in finet large language models. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), Advances in Neurol Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024b. URL http://papers nips. cctpaper_filess paper/2024/ htaahh//dddffff43333334466666661 bebbdd4-eebbeeeee Awwtract--ttoeeeeeeeeeeeeee html.. (Cited on 24)
Yu Peng, Zewen Long, Fangming Dong, Congyi Li, Shu Wu, and Kai Chen. Playing language game with Iims leads jailbreaking. CoRR, abs/2411.12762, 2024c. doi: 10.48550/ARX2 1472. URL
https://doi.org/ 11..8850//rriiv 24111.1.11111.........1111111111 12762. (Cited on 11)
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. Tn Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3419 3448, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https: //aclanthology. org/2022, emnlp-main.
otgg/202..mmlpmmainn 225. (Cited on 2, 4, 12, 21, 26, 29)
Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlishh Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatficld-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schieler, and Jared Kaplan. Discovering language model behaviors with model-writted evaluations. In Anna Rogers, Jordan (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 13387-13434, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/ //////222..2222.. 847. org.22... findings- acl. 847. (Cited on 2)
and Naoaki Okazaki
URL
53

---

Published in Transactions on Machine Learning Research (05/2025)
Perspective API. Perspective API Toxicity Categories. http://///rrbbooooooopppoepeeeep apiiatribbues-nddaaguuaaae 2024. (Cited on 3)
Willem Pienaar and Shahram Anver.
Rebuff:
Detecting
Prompt
Injection
Attacks.
2023. (Cited on 25)
Jeff Price. Red Alert: Are you ready for every airport manager's worst nightmare? Airport Magazine, 16, 2004. URL. http:://app.smmmniiccchooooooo wrw.aaarrrrr//rorottioio.o 106633227.. (Cited on 1)
Fanchao Qi, Yangyi Chen, Mukai Li, Zhiyuan Liu, and Maosong Sun. ONION: A Simple and Effective Defense Against Textual Backdoor Attacks. ArXiv, abs/2011. 2020. URL nttps://aptt .ttps///aaissemmtiiscccoo....
org/CorpusID:22711 22711866.. (Cited on 22)
Fanchao Qi, Yangyi Chen, Xurui Zhang, Mukai Li, Zhiyuan Liu, and Maosong Sun. Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer. ArXiu, abs/2110.07139, 2021a. URL sttpss///ap.ssmnniiiccscioee www.sseeaannntllleeee.e.ee wwwphhholarrrgg/rrrrrooee Srgg/CorpsSI::2330 23380707 (Cited on 19)
https://api.
Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger. In Annual Meeting of the Association for Computational Linguistics, 2021b. URL https://apl semant iishantccchhlar... wtw.ticchooooorrooorrrruuuur org/CorpusID:2355 2351 96099. (Cited
on 19)
Fanchao Qi, Yuan Yao, Sophia Xu, Zhiyuan Liu, and Maosong Sun. Turn the Combination Lock: Learnable Textual Backdoor Attacks via Word Substitution. In Annual Meeting of the Association for Computational Linguistics, 2021c. URL https: /appi.apppppaaaapee semanticscholar hatpillaa.rrrr/ppooeeee org/CorpusID: 2 235417102. (Cited on 19)
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Y Yu Chen, Ruoxi Jia, Prateck Mittal, and Peter Henderson. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! CoRR, abs 23100.3693, 2023. doi: 10.48550/ ARXIV 23310.03693. URL https://doi http:://doooooo////// 10. 18550/arXio 2310. 03693. (Cited on 4, 20)
Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety Alignment Should Be Made More Than Just Few Tokens Deep. 2024. URL (Cited 27)
Yao Qiang, Xiangyu Zhou, and Dongxiao Zhu. Hijacking Large Language Models via Adversarial In- Context Learning. ArXiv, (bs/2211.0999 2023. URL https://api.. semanticscholar (Cited on 4, 21)
https:://pi..smmnticccoooaar www.aarrrrr/oooooooeeee 265221 164.
Yao Qiang, Xiangyu Zhou, Salch Zare Zade, Mohammad Amin Roshani, Douglas Zytko, and Dongxiao Zhu. Learning to Poison Large Language Models During Instruction Tuning, 2024. URL https://ap/ semanticscholar. wwt.:///rossS::227777222222
(Cited on 4)
Baha. Rababah, Shang Tommy Wu, Matthew Kwiatkowski, Carson K. Leung, and Cuneyt Gurean Akcora. Sok: Prompt hacking of large language models. In 2024 JEFF International Conference on Big Data (BigData), pp. 5392 5401, 2024. doi: 10.1109/BigDaataaat Bttp///DiDaaa2233222....... 2082.10........ (Cited on 11)
Bhaktipriya Radharapu, Kevin Robinson, Lora Aroyo, and Preethi Lahoti. AART: AT-Assisted Red-Teamin with Diverse Data Generation for New LLM-powered Applications. In Mingxuan Wang and Imed Zitouni (eds.), Proceedings of the 2023 Conference on Empirical Melhods in Nalural Language Processing: EMNLP Association for Computational prg./223. emw.p-industtr..... 37. (Cited on 4, 13)
2023- Industry Track, Singapore, December 6-10, 2023, pp. Linguistics, 2023. URL
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is sccretly a reward model. arXiv preprint 17Xiv22305.18290, 2023. (Cited on 3)
Parijat Rai, Saumil Sood, Vijay Krishna Madisctti, and Arshdcop Bahga. GUARDIAN: A Multi-Tiered Defense Architecture for Thwarting Prompt Injection Attacks on LLMs. Journal of Software Engineering and Applications, 2024. URL https: //ppi semanticscholare wttp://o.....oo.ooooooooaaaaa11. (Cited on 25)
54

---

Published in Transactions on Machine Learning Research (05/2025)
Vyas Raina, Adian Liusie, and Mark Gales. Is LLM-as-a-Judge Robust? Investigating Universal Adversarial At- tacks on Zero-shot LLM Assessment. 2024. URL httpss//app: stt./pppi.smmaicscooolaaaaaaaa http://rrrr/roopsso///77770000/0 (Cited on 29)
Swaroop Indra Ramaswamy, Om Thakkar, Rajiv Mathews, Galen Andrew, H. B. McMahan, and Franccoise Beaufays. Training Production Language Models without Memorizing User Data. ArXiu, 0bbs/2009.1000 2020. URL https://aph semanticscholar www.llaa.rrrr/ooooo//ee www/CorpwsID::aaa.ccco 221811644. 9211 (Cited on 25)
Javier Rando and Florian Tramèr. Universal Jailbreak Backdoors from Poisoned Human Feedback. ArXio, abs/2311.14455, 2023. URL
wwwpaaniiisscoooooooooo/oooee : 265445004. (Cited on 4, 19)
Javier Rando and Florian Tramèr. RLHF Trojan Competition.
hutp://github.com//ee /ethz-
spylab/rlhf trojan _ompettiion, 2024a. (Cited on 2, 19)
Javier Rando and Florian Tramer. hwtmm//airandoo//saatt////B/B111
RLHF
Trojan Competition.
twit-
2024b. (Cited on 2, 19)
Abhinav Rao, Sachin Vashistha, Atharva Naik, Somak Aditya, and Monojit Choudhury. Tricking LLMs into Disobedience: Understanding, Analyzing, and Preventing Jailbreaks. arXiv preprint arXiv:2305. 14965, 2023. (Cited on 4)
James Reason. Human error. Cambridge University Press, 1990. (Cited on 25)
Traian Rebedea, Razvan Laurentiu Dinu, Makesh Narsinhan Sreedhar, Christopher Parisien, and Jonathan Cohen. NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails. In Conference on Empirical Methods in Natural Language Processing, 2023. URL https://api.
semanticscholar. wwtp://roo//oo../....... 264146531. (Cited on 22, 27)
Red Team. Red Team Wikipedia.
team. (Cited on 1)
Richard Ren, Steven Basart, Adam Khoja, Alice Gatti, Long Phan, Xuwang Yin, Mantas Mazeika, Alexander Pan, Gabriel Mukobi, Ryan H. Kim, Stephen Fitz, and Dan Hendrycks. Safetywashing: Do ai safety benchmarks actually measure safety progress?, 2024. URL https://arxiiv.rr//bbsstss 217922. (Cited on 29)
Maria Rigaki and Scbastian Garcia. A Survey of Privacy Attacks in Machine Learning ACM Comput. Surv., 56(4), nov 2023. ISSN 0360-0300, doi: 10.1145/3624010. URL on 15)
(Cited
Alexander Robey, Eric Wong, Hamed Hassani, and George J. Pappas. SmoothLLM: Defending Large Language Models Against Jailbreaking Allacks. ArXio, abs/2310.03684, 2023. URL https://app sea.aiicissmtitcchollllr org/CorpusID: 263671542. (Cited on 22, 27)
Christophe Ropers, David Dale, Prangthip Hansanti, Gabriel Mejia Gonzalez, Ivan Evtimov, Corinne Wong, Christophe Touret, Kristina Pereyra, Seohyun Sonia Kim, Cristian Canton- Pierre Andrews, and Marta Ruiz Costa-jussà, Towards Red Teaming in Multimodal and Multilingual Translation. ArXio, Abs/2240.11247,, 2024. URL https://api.semanticccollll
semanticscoolar .org www.ooogg/oooeeeeeoo CCrppIIDD::: 167311645. (Cited on 28)
Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bartoszcze, David Atanasov, Robie Gonzales, Subhabrata Majumdar, Carsten Maple, Hassan Sajjad, and Frank Rudzicz. Representation noising effectively prevents harmful fine-tuning on LLMs. 2024a. URL http:://ppp..smmmiiicccolaa ottpra/ticccccoooooooooo//oooooo ptt.ppusIIDD299986... 269982864. (Cited on 24, 27)
Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bartoszcze, Jan Batzner, Hassan Sajjad, and Frank Rudziez. Immunization against harmful fine-tuning attacks. 2024b. URL https:///ppi.smmntticchol... org/CorpusID: 268032044. (Cited on 24, 27)
Paul Röttger, Hannah Rose Kirk, Bertie Vidgen Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. XSTest: A Test Suite for Identifying Exaggerated Safoty Behaviours in Large Language Models. ArXio, abs/2308.0122 2023. URL https:/ //api. semanticscholar. htt.:ooola...re//roussi:::::::: 260378842. (Cited on 24, 29, 30)
55

---

Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose Published in Transactions on Machine Learning Research (05/2025)
Jessica Rumbelow and Matthew Watkins.
SolidGoldMaglkar (plus, prompt generation).
lesswrong.com wtt.m///wiiiiiioooooaaaaaapooopo
2023. (Cited on 12)
Mark Russinovich. Mitigating Skeleton Key, new type of generative AI jailbreak tech- /ttp://currillooo/////////////// generativeeillalllraa--tttuuuuiu 2024. (Cited on 4, 11)
nique.
Shiva Omrani Sabbaghi, Robert Wolfe, and Aylin Caliskan. Evaluating Biased Attitude Associations of Language Models in an Intersectional Context. Proceedings of the 2023 AAAI/ACM Conference on Al, (Cited on 2)
Ethics, and Society, 2023. URL
Vinu Sankar Sadasivan, Shoumik Saha, Gaurang Sriramanan, Priyatham Kattakinda, Atoosa Malemir Chegini, and Soheil Feizi Fast Adversarial Attacks on Language Models In One GPU Minute. ArXio, abs/2402.15570, 2024. URL http:////pp..mmmmticccchoooooeoe wwapchhollr.rr//oopooeeee wwwp/ooppssDoa///..... 267938703. (Cited on 4, 18)
Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Tupu, Eric Hambro, Aram H. Markosyan, Man- ish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Focrster, Tim Rocktaschel, and Roberta Raileanu. Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts. ArXie, abs/2400.16822, 2024. URL http:://app..smmmicccoooaa.rrrr
wwwps/ccccoaaaooooorooooeeeoe www.oorrr//roooo.o.ooo 268031 (Cited on 28)
Omar Sanseviero. Pickle Scanning, Safetensors, Social validation features.
twil-
ter.com wow.m///aseeiino/stttttttooeeeee / 1783//777773111114665533000 2024. (Cited on 6)
Opinions Do Language Models Reflect? ArXiv, abs/ 1754, 2023. URL
org/CorpusID:25738 257834040 (Cited on
William Saunders, Catherine Ych, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. CoRR, abs/ bbs/2206.00582 2022. doi: 10.48550/ARXYY 2206.05802. URL https: //doi. .eeeioi..ogg//// 0tt///1..8550///0
2206. 05802. (Cited on 23)
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language Models Can Teach Themselves to Use Tools. ArXiv, abs/ abs/2302.04761, 2023. URL https://apt http:::/app..emanniscchoo..rrr wwtpoolaa..rg/Coousst..... /Bg//CCppuIDD:256677342 256697342. (Cited on 7)
semantccscholar...
Jonas Schuett, Noemi Dreksler, Markus Anderljung, David McCaffary, Lennart Heim, Emma Bluemke, and Ben Garfinkel. Towards best practices in AGI safety and governance: A survey of expert opinion. ArXio, abss/230500753, 2023. URL.
org org/CCrpusID: 258676345. (Cited on 5)
Sander Schulhoff, Jeremy Pinto, Anaum Khan, Louis-Francee Bouchard, Chenglei Si, Svetlina Anati, Valen Tagliabue, Anson Kost, Christopher Carnahan, and Jordan Boyd-Graber. Igmore this title and HackAPrompt: Exposing systemic vulnerabilities LLMs through a global prompt hacking competition. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1945-4977, Singapore, December 2023. Association for Computational Linguistics. URL https://aclanthooogy. wrw.:/oogo.rrg///22//
doi: 10. 10.18653/vv /0663/vv/// wttpwv222vvavv 22023.cmmll-maan.. ott.t//tt////aaattooooo////////m main. 302. (Cited on 4, 6, 10, 11)
Barry Schwartz. Bing Webmaster Guidelines. httpss//rrrttegiiaaaccccc/c////t webmastergguiellies-44733 2024. (Cited on 17)
Leo Schwinn, David Dobre, Sophie Xhonneux, Gauthier Gidel, and Stephan Günnermann. Soft Prompt Threats: Attacking Safety Aligument and Unlearning in Open-Source LLMs through the Embedding Space. Arxiv, abs/ hts/2402.09066 2024. URL https::/apll 4)
(Cited on
Rico Sennrich, Jannis Vamvas, and Alireza Mohammadshahi. Mitigating Hallucinations and Off-target and Contrastive Decoding. ArXiu, (Ciiee (Cited on 24)
Machine Translation with
2023. URL. https: http://p..eaaaatcccoooooorrrrrrp
56

---

Published in Transactions on Machine Learning Research (05/2025)
Erfan Shayegani, Md Abdullah Al Mamun, Yu Fu, Pedram Zaree, Yue Dong, and Nael B. Abu-Ghazalch. Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks. ArXio, abs/2310.108444 (Cited on 6)
2023. URL https://apil semanticscholarttttteeeeeeecceee http:://aai..mmttticcccolllll wwwp:ooaaarrrr/ooooo//eoo http://o//ooooSS:2/4111111
Guangyu Shen, Siyuan Cheng, Kai xian Zhang, Guanhong Tao, Shengwei An, Lu Yan, Zhuo Zhang, Shiqing Ma, and Xiangyu Zhang. Rapid Optimization for Jailbreaking LLMs via Subconscious Exploitation and Echopraxia. ArXio, abs//400.005667, 2024. URL / semantcschooaar .rg :ttp://rruS:::::::00 267547566. (Cited on 4)
Lujia Shen, Shouling Ji, Xuhong Zhang, Jinfeng Li, Jing Chen, Jie Shi, Chengfang Fang, Jianwei Yin, and Ting Wang. Backdoor Pre-trained Models Can Transfer to All. Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, 2021. URL https://api. semanti semanticscholar wwt.rcccoooo.ooooorousoosse org/CorpusID: 240354696 (Cited on 19)
Sheng Shen, Le Hou, Yan-Quan Zhou, Nan Du, S. Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, Tu Vu, Yuexin Wu, Wuyang Chen, Albert Webson, Yunxuan Li, Vincent Zhao, Hongkun Yu, Kurt Keutzer, Trevor Darrell, and Denny Zhou. Mixturo-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models. 2023a. URL https: semanniccma wrt. :
(Cited on 16)
Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. Large Language Model Alignment: A Survey, ArXie, abs/2309.1502 2023b. URL
httptttiiiiccooaooooooroooeeeeee org//orppstDD. 262.p4448022282281..11 262824801. (Cited on 3)
Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 'Do Anything Now': Characterizing and Evaluating ue-The-Wild Jailbreak Prompts on Large Language Models. CoRR abs/2308. 03822,, 2023c. (Cited on 2)
Xinyue Shen, Zeyuan Johnson Chen, Michael Backes, Yun Shen, and Yang Zhang. 'Do Anything Now": Characterizing and Evaluating In-The-Wiih Wild Jailbreak Prompts on Large Language Models. ArXiv, abs/2300 0te//3388.03825,, 2023d. URL https: //api. sematiicccsatt www.oogg//ooo.... brgg/CorpusID:2600442222
(Cited on 4, 10)
Xuan Sheng, Zhaoyang Han, Piji Li, and Xiangmao Chang. A survey on backdoor attack and defense in natural language processing. In 2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS), pp. 809 820. IEEE, 2022. (Cited on 19)
Xuan Sheng, Zhicheng Li, Zhaoyang Han, Xiangmao Chang, and Piji Li. Punctuation Matters! Stealthy Backdoor Attack for Language Models. In Fei Liu, Nan Duan, Qingting Xu, and Yu Hong (eds.), Natural Language Processing and Chinese Computing 12th National CCF Conference, NLPCC 2023, Foshan, China, October 12-15, 2023, Proceedings, Part volume 14302 of Lecture Notes in Computer Science, pp. 524 536. Springer, 2023. doi: 978- 3- 001---31------- 1_41. (Cited on 19)
1) 41. URL https://doi.onn http:///oo.oo//1..0//0/
Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, Lewis Ho, Divya Siddarth, Shahar Avin, William T. Hawkins, Been Kim, lason Gabriel, Vijay Bolina, Jack Clark, Yoshua Bengio, Paul Francis Christiano, and Allan Dafoe. Model evaluation for extreme risks. ArXiv, abs/2305.1522 15324, 2023. URL http:://ipi.semaiiiisssseoee
org/ http//CoouDDDDBBSSSS.
(Cited on 5)
R. Shokri, Marco Stronati, Congzhong Song, and Vitaly Shmatikov. Membership Inference Attacks Against Machine Learning Models. 2017 JEEE Symposium on Securily and Privucy (SP), pp. 18, 2016. URL htt.://pp...mmmniiicccccoo..aaa. wwwpccholarrrrgee/eeoooooo wtt.oorr///ooo//// :anisIIDD:1488880 10488675. (Cited on 14)
Manli Shu, Jiong Wang, Chen Zhu, Jonas Geiping, Chaowei Xiao, and Tom Goldstein. On the Exploitability of Instruction Tuning. ArXiv, abs/ //////206.11999 2023. URL https: //a:: semaanticschhlaarrrr//orrrssssse 259309096. (Cited on
57

---

Published in Transactions on Machine Learning Research (05/2025)
Aradhana Sinha, Ananth Balashankar, Ahmad Beirami, Thi Avrahami, Jilin Chen, and Alex Beutel. Break it, Tmitate it, Fix it: Robustness by Generating Human-Like Attacks. ArXiv, abs/2310.16955, 2023. URL semanticccact httptt/scsccooooooogo/rrrpssssps
(Cited on 28)
Chawin Sitawarin, Norman Mu, David Wagner, and Alexandre Araujo. PAL: Proxy-Guided Black-
Attack on Large Language Models. ArXiv, has//200.009674 2024. URL https://ap1: http:///pp.smmmniccccol....s
org/CorpusID:26766 267682038, (Cited on 4)
Ram Shankar Siva. Kumar.
A Few Useful Lessons
about
A1
Red Teaming.
httpaabbeeujjjjji_e-------- =??t=2025, 2023a. (Cited on 6)
Ram Shankar Siva Kumar. Microsoft AI Red Team building future of safer AL. http:://www..iccooof..comm com.s//oooccommn--e-ee 2023b. (Ciled on 2)
Sesecurrtyy/log/2000
Victoria Smith, Ali Shahin Shamsabadi, Carolyn Ashurst, and Adrian Weller. Identifying and Mitigating Privacy Risks Stemming from Language Models: A Survey. ArXiv, abs/2310.01424, 2023. URL https:
//api..smmnticcchollar httpr//ciiicccccooooorr/rrrrrpae 2636087722 (Cited on 25)
Saleh Soltan, Shankar Ananthakrishnan, Jack FitzGerald, Rahul Gupta, Wael Hamza, Haidar Khan, Charith Peris, Stephen Rawls, Andy Rosenbaum, Anna Rumshisky, Chandana Satya Prakash, Mukund Sridhar, Fabian Triefenbach, Apury Verma, Gökhan Tür, and Prem Natarajan. AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model. CoRR, abs/ 22208.01448, 2022. doi: 10.48550/ ARXIV. 2208.01448, URL https: / wtw.ooooo///ooo//ooo//o//oooo/oo wtw.//.///////.. 1w..1....///.0 48550/arXiv. 2208 01448. (Cited on 6)
Gowthami Somepalli, Vasu Singla, Micah Goldbhum, Jonas Geiping, and Tom Goldstein. Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models. 2023 IREE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pp. 6048 6058, 2022. URL https: seannticaanic hotpo/larrrrr//ooooe orgg/CorpuII:: :00opuu11:224366644 254366634. (Cited on 14)
Congzheng Song and Ananth Raghunathan. Information Leakage in Embodding Models. Proceedings of the 2020 ACM SIGSAC Conference on Computer and Commnunications Securily, 2020. URL https: //apii wttps/coooaa..orrrouupuuslll
(Cited on 16)
Shuang Song, Kamalika Chandhuri, and Anand D. Sarwate. Stochastic gradient descent with differentially private updates. 2013 TEFE Global Conference on Signal and Information Processing, pp. 245 245-248, 228,, 2013. URL https://ap1: semantccshoolar orgg/CorpssID: 8085841. (Cited on 25)
Murugiah Souppaya, Karen Scarfone, and Donna Dodson. Secure software development framework (ssdf) version 1.1. NIST Special Publication, 800:218, 2022. (Cited on 31)
Logan Stapleton, Jordan Taylor, Sarah Fox, Tongshuang Sherry Wu, and Haiyi Zhu Seeing Seeds Beyond Weeds: Green Teaming Generative AI for Beneficial Uses. ArXio, abs/2306.03097, 2023. URL https:
//api. semmaniccanti.c httpchoooaar.rrr/oouuss--::::.:a //:CCopss:::559888580 (Cited on 3, 29)
Asa Cooper Stickland, Alexander Lyzhov, Jacob Pfau, Salsabila Mahdi, and Samuel R. Bowman. Steering Without Side Effects: Improving Post-Deployment Control of Language Models. 2024. URL https: //api. sewantisschooollrr Drrg/CorpusI::: 2EW.ID::777033 270703306. (Cited on 27)
Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Zheng Liu, Yixin Liu, Yijue Wang, Zhikun Zhang, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, Chun-Yan Li, Eric P. Xing, Furong Huang, Haodong Liu, Heng Ji, Hongyi Wang, Huan Zhang, Huaxiu Yao, Manolis Kellis, Marinka Zitnik, Meng Jiang, Mohit Bansal, James Zou, Jian Pei, Jian Liu, Jianfeng Gao, Jiawei Han, Jieyu Zhao, Jiliang Tang, Jindong Wang, John Mitchell, Kai Shu, Kaidi Xu, Kai-Wei Chang, Lifang Hc, Lifu Huang, Michael Backes, Neil Zhenqiang Gong, Philip S. Yu, Pin-Yu Chen, Quanquan Gu, Ran Xu, Rex Ying, Shuiwang Ji, Suman Sekhar Jana, Tian-Xiang Chen, Tianming Liu, Tianying Zhou, William Wang, Xiang Li, Xiang- Yu Zhang, Xiao Wang, Xingyao Xie, Xun Chen, Xuyu Wang, Yan Liu, Yanfang Ye, Yinzhi Cao, and Yue Zhao. TrustLLM: Trustworthiness in Large Language Models. ArXiv, abs/2401.0556)) 055611, 2024. URL https: //api. www.antccccaarrrooo/orrossoooooo 266933333. (Cited on 5)
58

---

Published in Transactions on Machine Learning Research (05/2025)
Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A Simple and Effective Pruning Approach for Large Language Models. ArXiv, abs/2306.11 11695, 2023. URL 259203115. (Cited on 27)
Xuchen Suo. Signed-Prompt: A New Approach to Prevent Prompt Injection Attacks Against LLM-Integrated Applications. ArXiv, abs/ 24000400.,761,, 2024. URL (Cited on 27)
Alex Tamkin, Miles Brundage, Jack Clark, and Deep Ganguli. Understanding the capabilities, limitations, and societal impact of large language models. arXiv preprinl arXiv:2102 01.:000 2021. (Cited on 5)
Liyan Tang, Z. Sun, Betina R.S. Idnay, Jordan G. Nestor, Amani Soroush, Pablo Adolfo Flias, Z. P. Xu, Y. Ding, Greg Durrett, Justin F. Rousscau, Cindy Weng, and Yalci Peng. Evaluating large language models on medical evidence summarization. NPJ Digital Medicine, 6, 2023. URL https://ppi.smmaniicccholl.. semantisccoola.r org/CorpusID: 2582997177. (Cited on 6)
Xiangru Tang, Qiao Jin, Kunlun Zhu, Tongxin Yuan, Yichi Zhang, Wangchunshu Zhou, Meng Qu, Yilun Zhao, Jian Tang, Zhuosheng Zhang, Arman Cohan, Zhiyong Lu, and Mark B. Gerstein. Prioritizing Safeguarding Over Autonomy: Risks of LLM Agents for Science. ArXiv, abs/2402.04247,, 2024. URL https://api.. http:/ppp..smnnniisschhoolrrr wwtp:/ccooaaa..orrooupppssseeeee 267499916 (Cited on 21)
Simone Tedeschi, Felix Friedrich, Patrick Schramowski, Kristian Kersting, Roberto Navigli, Huu Nguyen, and Bo T.i. ALERT: Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming, ArXiv, abs/2404.05666,, 2024. URL https://api. https://app..smantiiccoo... hotpr//scccccooooooo//rrrppooooo 2600rppsID::211490000 269149567. (Cited on 3, 29)
Guardrails AI. Guardrails AI: Adding guardrails large language models. https://github.com guardrails-aile ailllls-aigga 2024. (Cited on 22, 27)
The White House. Exccutive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence. https//wwwwwwwithhosssee. gov /ow////ieiiig-ooooo roooing-rroo/oooooooo tooomppon ppepsddeeeednie/e cal-actiions/0220000 http://22222/////////eeeeeeeee /00e aateeeaaf--cccccaaadd-tttottr-rr
anntlloppeett-t---e------a use-uf-autt wwwtiiiiial-intelllleeeee
2023. (Cited on 2)
Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D. Manning, and Chelsea Finn. Fine-tuning Language Models for Factuality. ArXiv, h/s/2211.088A 2023. URL https://api. semanticscholar. wwtpsschhooarroo/ooooooooo org/CorpusID: 265158181. (Cited on 24)
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaune Tample. ILLMAA: Open and Effiicentt Foundation Language Models. ArXie, (bs//2332.19777, 2023. URL https////ap.semnnicccooooo.... semmantiiiccolaarr.rr////rooss/ 257219404. (Cited on 20)
Sam Toyer, Olivia Watkins, Ethan Adrian Mendes, Justin Svegliato, Luke Bailey, Tiffany Wang, Isaac Ong, Karim Elmaaroufi, Pieter Abbeel, Trevor Darrell, Alan Ritter, and Stuart Russell. Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienma, Austria, May 7-11, 2024. OpenReview.ng .net, 2024. URL http:://pppeeviieeeeeeffooornnne
(Cited on 2)
Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter, and Thomas Ristenpart. Stealing Machine Learning Models via Prediction APIs. In USENIX Security Symposium, 2016. URL htttps://ppi stt./ppps.smmmnicccooo...aaa org/CorpusID:2284552. (Cited on 14)
Jean Marie Tshimula, Xavier Ndona, D'Jeff K. Nkashama, Pierre-Martin Tardif, Froduald Kabanza, Marc Frappier, and Shengrui Wang. Preventing jailbreak prompts as malicious tools for cybercriminals: A cyber defense perspective. ArXiv, abs/2411. 166442, 2024. URL https: //api. semanticscholar. htt.s/cccccaaa..rrr///ooossooo org/CorpusID: 274280511. (Cited 3)
59

---

Published in Transactions on Machine Learning Research (05/2025)
Alexander Matt Turner, Lisa Thiergart, David S. Udell, Gavin Leech, Ulisse Mini, and Monte Stuart MacDi- armid. Activation Addition: Steering Language Models Without Optimization. ArXie, abs/2308. 10248,
2023. URL https://app: http:/ppp.mmmnnicccccoaoaar..rra http://ss.......o.oooooooss//aaa (Cited on 18)
Usage Policy Anthropic. Anthropic Usage Policies. http:://wwwwaatthhrocccooo/////s /aup, 2023. (Cited on 3)
Usage Policy OpenAT. OpenAT Usage Policies.
2023. (Cited on 3)
Thomas Vakili and Hercules Dalianis. Using Membership Inference Attacks to Evaluate Language Modeling Fails for Pseudonymizing Data. In Proceedings of the 24th Nordic Conference Computational Linguistics (NoDaLiDa), pp. 318-323, 2023. (Cited on 15)
Matheus Valentim, Jeanette Falk, and Nanna Inie. Hacc-Man: An Arcade Game for Jailbreaking LLMs. 2024. URL
(Cited on 2)
Laurene Vaugrante, Francesca Carlon, Malua Monke, and Thilo Hagendorff. Compromising honesty and harmlessness in language models via deception allacks. 2025. URL https: //aps://pi..a/ semanticscholar.. org/ CorpusSD:276287940. (Cited on 29)
Pranshu Verma and Will Oremus. These lawyers used ChatGPT to save time. They got fired and fined. http://wwww.aihitttoooosssttttt. cctp:/ccccollaa///2///////cccccp
2023. (Cited on
2)
Joe Vest and James Tubberville. Red Team Development and Operations A practical Guide. Independently Published, 2020. (Cited on 1)
Bertie Vidgen, Adarsh Agrawal, Ahmed M. Ahmed, Victor Akinwande, Namir Al-nuaimi, Najla Alfaraj, Flie Alhajjar, Lora Aroyo, Trupti Bavalatti, Borhane Blili-Hamelin, Kurt D. Bollacker, Rishi Bomassani, Marisa Ferrara Boston, Sim'eon Campos, Kal Chakra, Canyu Chen, Cody Coleman, Zacharie Delpierre Coudert, Leon Derczynski, Debojyoti Dutta, Ian Eisenberg, James R. Ezick, Heather Frasc, Brian Fuller, Ram Gandikola, Agasthya Gangavarapu, Ananya Gangavarapu, James Gealy, Rajat Ghosh, James Goel, Usman Gohar, Sujata Goswami, Scott A. Hale, Wiebke Hutiri, Joseph Marvin Imperial, Surgan Jandial, Nicholas C. Judd, Felix Juefei-Xu, Foutse Khomh, Bhavya Kailkhura, Hannah Rose Kirk, Kevin Klyman, Chris Knotz, Michael Kuchnik, Shachi H. Kumar, Chris Lengerich, Bo Li, Zeyi Liao, Eileen Peters Long, Victor Lu, Yifan Mai, Priyanka. Mary Mammen, Kelvin Manyeki, Sean McGregor, Virendra Mehta, Shafee Mohammed, Emanuel Moss, Lama Nachman, Dinesh Jinenhally Naganna, Amin Nikanjam, Besmira Nushi, Luis Oala, Iftach Orr, Alicia Parrish, Çigdem Patlak, William Pietri, Forough Poursabzi- Sangdeh, Eleonora Presani, Fabrizio Puletti, Paul Röttger, Saurav Sahay, Tim Santos, Nino Scherrer, Alice Schoenaner Scbag, Patrick Schramowski, Abolfazl Shahbazi, Vin Sharma, Xudong Shen, Vamsi Sistla, Leonard Tang, Davide Testuggine, Vithursan Thangarasa, Elizabeth Anne Watkins, Rebecca Weiss, Christoper A. Welty, Tyler Wilbers, Adina Williams, Carole-Jean Wu, Poonam Yadav, Xianjun Yang, Yi Zeng, Wenhui Zhang, Fedor Zhdanov, Jiacheng Zhu, Percy Liang, Peter Mattson, and Joaquin Vanschoren. Introducing v0.5 of the AI Safety Benchmark from MLCommons. ArXiv, 2024. URT. https://api: https://api..mmmnticchhhei :tt.://aa..rggccopps..::200:00 269214329. (Cited on 3)
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing NLP. In Proceedings of the 2019 Conference on Empérical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 2153 2162, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/ 1..18553/11/D1-12200 1221. URL
(Cited on 13)
Eric Wallace, Tony Zhao, Shi Feng, and Sameer Singh. Concealed Data Poisoning Attacks on NLP Models. In North American Chapter of the Association for Computational Linguistics, 2021. URL https://api.
semanticscholar. Sammatiisssooorrrorrr///oooo/:s: 2333330114 (Cited on 19)
Eric Wallace, Kai Xiao, Reimar H. Leike, Lilian Weng, Johannes Heidocke, and Alex Beutel. The Instruction Hicrarchy: Training LLMs to Prioritize Privileged Instructions. 2024. URL https: //ppi. semanticscholar. org//ooppuSDD:22924444.. (Cited on 24, 27)
60

---

Published in Transactions on Machine Learning Research (05/2025)
Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. Poisoning Language Models During Instruction Tuning. ArXiv, abs/2305.009444, 2023. URL https://api. wwwpattiiiscoooooorooorrroee 258426823.
(Cited on 4, 19)
Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and B. Li. Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Langnage Models. ArXiv, abs/2111.028400 2021. URL https://apl semanticscholar, org/CorpusID::22275771) (Cited on 24)
Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zi-Han Lin, Yuk-Kit Cheng, Sanmi Koyejo, Dawn Xiaodong Song, and Bo Li. DecodingTrung Trust: A Comprehensive Assessment of Trustworthiness in GPT Models. ArXiv, 2ts/2306.116999 2023a. URL https://api. semanticscholar. wwtp:/oooo/ooo:2555000 259202722. (Cited on 3, 5, 6, 29)
Chaoqi Wang. Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints. ArXiv, 0ts/2300.1622444 2023b. URL (Cited on 23)
/ttts://api..smmnticcee http:://.....ooooootooooooroo//s 263142109
Haoran Wang and Kai Shu. Backdoor Activation Attack: Attack Large Language Models using Activation
Steering for Lafeet-AAignmentt ArXie, abs/2311.09433,, 2023. URL https:
org/
CorpusID:265220922 (Cited on 4, 18)
Jiong Wang, yang Liu, Keun Hee Park, Muhao Chen, and Chaowei Xiao. Adversarial Demonstration Attacks on Large Language Models. ArXiv, abs/2305.1440 2023c. URL
org/CorpusID::58866 258865339. (Cited on 4)
Jiong Wang, Jiazhao Li, Yiquan Li, Xiangyu Qi Junjie Hu, Yixuan Li, Patrick Drew McDaniel, Muhao Chen, Bo Li, and Chaowei Xiao. Mitigating Fine-tuning Jailbreak Attack with Backdoor Enhanced Alignment. Ar Xiv, abs/2442.14968, 2024a. URL https: //api. cisrannisccholrr...o oot.oooooor///////r//s11:::::::0 24, 27)
orgg/CorpusD::268877555 (Cited on
Lei Wang, Chengbang Ma, Xueyang Feng, Zeyu Zhang, Hao ran Yang, Jingsen Zhang, Zhi- Yang Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji rong Wen. A Survey on Large Language Model based Autonomous Agents. ArXiv, abs/2308.11 11432, 2023d. URL https://api..emmnticchoot semanticscholar. org/CorpusID: 2012222122444 064713. (Cited on 21)
Meng Wang, Ningyu Zhang, Ziwen Xu, Zekun Xi, Shumin Deng, Yunzhi Yao, Qishen Zhang, Linyi Yang, Jindong Wang, and Huajun Chen. Detoxifying Large Language Models via Knowledge Editing. ArXis, 0ts/2200311472,, 2024b. URL https://apl.
268553537. (Cited on 28, 29)
Yihan Wang, Zhouxing Shi, Andrew Bai, and Cho-Jui Hsich. Defending L.Mss against Jailbreaking Attacks via Backtraanlation. ArXiu, abs/2402.16458, 2024c. URL https://api. semanticscoolar. hatpaatttssscsooooooor/////rooo 268032484. (Cited 22, 27)
Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning Large Language Models with Human: A Survey. ArXiv, abs/ abs/2207.12966, 2023e. URL https: semanticscholar org/CorpusID. st 260356605. (Cited on 3)
Ziqiu Wang, Jun Liu, Shengkai Zhang, and Yang Yang. Poisoned LangChain: Jailbreak LLMs by LangChain. wwwanticscoolarrrrreeeeoeoo
2024d. URL http:://ppp...eeaaattntccoororrr orgp/Corpu1D:22773311111 (Cited on 17)
Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How Does LLM Safety Training Fail? ArXio, abs/2307.022 024833, 2023a. URL httpp:://ppl sttp:/iiii..mmmiiicccccchllala org/CorpusID:2553422 (Cited on 4, 11)
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent Abilities of Large Language Models. Trans. Mach. Learn. Res., 2022, 2022. URL htttss///pperrviieeeee//rrrrrrr/
(Cited on
61

---

Published in Transactions on Machine Learning Research (05/2025)
Junyin Wei, Yicheng Zhang, Zhe Zhou, Zhou Li, and Mohammad Abdullah Al Faruque. Leaky DNN: Stealing Deep-Tearning Model Secret with GPU Context-Switching Side-Channel 2020 50th Annual IEEE/ /IFI International Conference on Dependable Systems and Networks (DSN), pp. 125 137, 2020. URL attps://aptt semanticscholar wwwaaaarrrg////ooaaaeeee wrggCorpusSD: 220939286. (Cited on 17)
Lingxiao Wei, Yannan Liu, Bo Luo, Yu LI, and Qiang Xu. Know What You See: Power Sede-Channel Attack on Convolutional Neural Network Accelcrators. Proceedings of the 34th Annual Computer Security Applications Conference, 2018. URL https://apt semanticscholar, hwwpeschoororor/oooooooooooo atg/CorppsID:3920550 (Cited on 17)
Zeming Wei, Yifei Wang, and Yisen Wang. Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations. ArXiv, abs/2310.06387,, 2023b. URL https: //api. semanticscholar / CorpuSID: (Cited on 4)
Laura Weidinger, John F. J. Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zachary Kenton, Sande Minnich Brown, William T. Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William S. Isaac, Sean Legassick, Geoffrey Irving, and lason Gabriel. Ethical and social risks of harm from Language Models. ArXiv, (1s/2211.04359, 2021. URL https: https://api.ssemnti/t semanticscholar htt.ccccooao..oooorrrroeeo org/CorpusID:
244954639. (Cited on 2, 6, 28)
Laura Weidinger, John F J Mellor, Bernat Guillén Pegueroles, Nahema Marchal, Ravin Kumar, Kristian Lum, Canfer Akbulut, Mark Diaz, A. Stevie Bergman, Mikel D. Rodriguez, Verena Rieser, and William Isaac. STAR: SocioTechnícal approach to red teaming language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 21516-21531 Miami, Florida, USA, November 2024. Association for Computational URL https://aclanthologyy wrgg/024/ emnlp-main.
Linguistics. doi: 10.18653 (10055/vv// hetp//2..mmllmmiin....... 1200/. (Cited on 30)
Roy Weiss, Daniel Ayzenshteyn, Guy Amit, and Yisroel Mirsky. What Was Your Prompt? A Remote Keylogging Attack on Al Assistants. 2024. URL (Cited on 4)
Jiaxin Wen, Pei Ke, Hao Sun, Zhexin Zhang, Chengfei Li, Jinfeng Bai, and Minlie Huang. Unveiling the Implicit Toxicity in Large Language Models. In Houda Bouamor, Juan Pino, and Kalika Bali (ods.), Proceedings of the 2023 Conference on Empiricul Methods in Natural Language Processing, pp. 1322 1338, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emm emm.lp- main.84. URL https://aclaathoology vrgg/223..e mainp 84. (Cited on
Nevan Wichers, Carson E. Denison, and Ahmad Beirami. Gradient- Based Language Model Red Teaming. ArXiu, abs/2401.16656, 2024. URL https://aat semanticscholar 4, 20, 21)
ssmaattiiccoolar..ooo//oopossooo wtgp/CorpuID::2732200311 (Cited on
David Gray Widder, Sarah West, and Meredith Whittaker. Open (For Business): Big Tech, Concentrated Power, and the Political Economy of Open AL. SSRN Electronic Journal, 2023. URL https://api. semanticscholar http://cccooaaaoorg//orrrroooooa 266006601D::226670311 (Cited on 2)
Yotam Wolf, Noam Wies, Yoav Levine, and Amnon Shashua. Fundamental Limitations of Aligument in Large Language Models. ArXit, abs/2304.11092, 2023. URL https://apt: www.nticschoollrrrrmrmom. httpss//appi..emntiicccoooor 258291526 (Cited on 29)
org/CorpusID: Dorea
Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, Mingli Zhu, Ruotong Wang, Li Liu, and Chaoxiao Shen. BackdoorBench: A Comprehensive Benchmark and Analysis of Backdoor Learning. ArXie, abs/2400.11000 2024a. URL (Cited on 29)
267301 142.
Xi Wu, Matthew Frodrikson, Somesh Jha, and Jeffrey F Naughton. A methodology for formalizing model- inversion attacks. In 2016 IEEE 29th Computer Security Foundations Symposium (CSF), pp. 355-370. TEEE, 2016. (Cited on 4, 15)
62

---

Published in Transactions on Machine Learning Research (05/2025)
Yuanwei Wu, Xiang Li, Yixin Liu, Pan Zhou, and Lichao Sun. Jailbreaking GPT-4V via Self-Adversarial Attacks with System Prompis. CoRR, abs/2311 .09127, 2023. doi: 00.45500/AXXXVV2211..0011 URL https://doi.org// 48550/arXiv 2311. (Cited on 4)
Yuhao Wu, Franziska Roesner, Tadayoshi Kohno, Ning Zhang, and Umar Iqbal. SecGPT: An Execution Isolation Architecture for LLM-Based Systems. 2024b. URL https://apls semanticschollrr.... httpatttsssccollooooooorrrrueeee org/CorpusID: 268296997. (Cited 31)
Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhemmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. OS-Copilot: Towards Generalist Computer Agents with Self-mprrvvment. ArXie, abs/2442.07446, 2024c. URL bttps://apti semanticscholar wttpetticccccoaooooorrroirroeeee 267626995. (Cited on 7)
Chong Xiang, Tong Wu, Zexuan Zhong, David A. Wagner, Danqi Chen, and Prateck Mittal. Certifiably robust RAG against retrieval corruption. CoRR, (bss/2405115555 2024a. doi: 10.48550/ St..0/ARXIV...22 1505. URL https://doi.org 10. 48550/arXiv 2405. 15556. (Cited on 18)
Yun Xiang, Zhuangzhi Chen, Zuohui Chen, Zebin Fang, Haiyang Hao, Jinyin Chen, Yi Liu, Zhefu Wu, Qi Xuan, and Xiaoniu Yang. Open DNN Box by Power Sido-Channel Attack. IEEE Transactions on Circuits and Systems II: Eapress Briess, 67:2717 2721, 2019. URL https://api semanticscholaat Senecholar.org/oorreoo org/CorpuSID: 198229526. (Cited 17)
Zhen Xiang, Fengqing Jiang, Zidi Xiong, Bhaskar Ramasubramanian, Radha Poovendran, and Bo Li. BadChain: Backdoor Chain-of-Thoughle Prompting for Large Langnage Models. ArXiu, abs/2401.12242,, 2024b. URL https://api. http://ppp.ssmmttiiicsssiesses semanticscholar wtt.sechhoo...oorrrrrruuoueeeeee orrg/CorpssI::25555 2670944877 (Cited on 4, 17)
Yijia Xiao, Yiqiao Jin, Yushi Bai, Yue Wu, Xianjun Yang, Xiao Luo, Wenchao Yu, Xujiang Zhao, Yanchi Liu, Haileng Chen, Wei Wang, and Wei Cheng, Large Language Models Can Be Good Privacy Protection Learners. ArXis, (Cited on 27)
2023. URL https://aplt http:/ppp...mmatccccccoorrrrrrrr
Zeguan Xiao, Yan Yang, Guanhua Chen, and Yun Chen. Tastle: Distract Large Language Models for Automatic Jailbreak Autakk ArXiv, abs/2403.08424, 2024. URL https://api. o 268379559. (Cited on 26)
Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixnan Huang, Laxi He, Boyi Wei, Dachong Li, Ying Shong, Rnoxi Jia, Bo Li, Kai Li, Danqi Chen, Peter Henderson, and Pratook Mittal. SORRY-Bench Systematicaaly Evaluating Large Language Model Safety Refusal Behaviors. 2024a. URL https://apt. semanticscholaatt ott.ammiiiicct....rrrrrrrrooooao
(Cited on 29, 31)
Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. Defending ChatGPT against jailbreak attack via self-reminders. Nature Machine Intelligence, 5:1486-1485 2023. URL https: //api. swwanti httpss//app...mmmiiiccoooooooo. wwwaaa..rrg/oppooeeeeee
(Cited on 22, 27)
Yueqi Xie, Minghong Fang, Renjie Pi, and Neil Zhenqiang Gong. GradSafe: Detecting Jailbreak Prompts for
LLMs via Safety-Critical Gradient Analysis. 2024b. URL http::://app.smmmttissssheeeeeee CawaaarrorrCCoousoriteeee 267770418. (Cited on 28)
Guo Xingang, Fangxu Yu, Huan Zhang, Lianhui Qin, and Bin Hu. COLD-Attack: Jailbreaking LLMss with Stealthiness and Controllability. ArXiv, abs/2402.08679, 2024. URL http:://a.p...spsscssiiieee org/ CorpusID: 26663721.. (Cited on 4, 21)
Chen Xiong, Xiangyu Qi, Pin-Yu Chen, and Tsung-Yi Yi Ho. Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attacks. 2024. URL CrrpuIDD:200123431 (Cited on 27)
Jiashu Xu, Mingyu Derek Ma, Fei Wang, Chaowei Xiao, and Muhao Chen. Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuming for Large Language Models. ArXio, abs/2305.14710, 2023a. www.aaoooooooooo//rrrrroo/////--
URL https://apts //api. semanticscholar. orgp/CorpusID::58866211 (Cited on 4, 19)
63

---

Published in Transactions on Machine Learning Research (05/2025)
Nan Xu, Fei Wang, Ben Zhou, Bangzheng Li, Chaowei Xiao, and Muhao Chen. Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinking. ArXiv, abs/2311.09827,, 2023b.
URL https: //ppi. ott.aatttssttt...ooooooorooooooa wtgg/oorpuSD::2622233333
(Cited on 4, 13)
Yue Xu and Wenjie Wang. LinkPrompt: Natural and Universal Adversarial Attacks on Prompt-based Language Models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapler of the Associalion for Compulational Linguislics: Human Language Technologies (Volume 1: Long Papers), pp. 6473 6486, Mexico City, Mexico, June 2024. Association for Computational Linguistics. URL pttps://aclanthool..e wwg.///........././ 2ere244aacllooogg 360. (Cited on 13)
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Poovendran. SafeDe- Defending against Jailbreak Attacks via Swfety-Aaare Decoding. ArXiv, abs/2402.08963, 2024a.
coding:
URL https://app semanticscholar.
wwtpssholarr.og/oorpusssss// org/Coopuu1D:227658823
(Cited on 22, 27)
Zihao Xu, Yi Liu, Gelei Deng, Yuekang Li, and Stjepan Picek. LLM Jailbreak Attack versus Defense Techniques A Comprehensive Study. 2024b. URL https://apl http:////aaissmmiiiscsiiiiio wew.aatttccccoooo...oo///ooooooo orgg/CorpusD::2277792444 (Cited
semanti
on 22)
Jiaqi Xue, Mengxin Zheng, Ting Hua, Yilin Shen, Yepeng Liu, Ladislau Bölöni, and Qian Lou. TrojLLM: A Black- box Trojan Prompt Attack on Large Language Models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Newral Information Processing Systems, volume 36, pp. 65665-65677. Curran Associates, Inc., 2023. URL https://proceeediggs neurips. cc/ cc/paper_filees http:///......eeeeeeoeeeeeeeeeea orr.aa/00000000e000aaaaaaaaa-rr-
pdf. (Cited on 4, 13)
Jun Yan, Vikas Yadav, SHIYANG LI, Lichang Chen, Zheng Tang, Hai Wang, Vijay Srinivasan, Xiang Ren, and Hongxia Jin. Backdooring Tustruction-Tuned Large Language Models with Virtual Prompt Injection. 2023. URL https: //api.
(Cited on 4)
Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin, and Xia Hu. Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond. ArXiv, abs abs/2304.33712, 2023a. URL https: / /api..ppp..pp semanticscholar. org wrgg/CorpuSI::25... 258331833 (Cited on 3)
Wenkai Yang, Lei Li, Zhiyuan Zhang, Xuancheng Ren, Xu Sun, and Bin He. Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models. ArXie, abs/2103.15543, 2021a. URL https://api. semantischoolllaat org/ wttgg//oossSD:::2 2t2p::II:I:222111113 232404131. (Cited on 19)
Wenkai Yang, Yankai Lin, Peng Li, Jie Zhou, and Xu Sun. Rethinking Stealthiness of Backdoor Attack against NLP Models. In Annual Meeting of the Association for Computational Linguisties, 2021b. URL
org/ wwtrgg/CrpuSD::::: 23145909:33444933... (Cited on 19)
Wenkai Yang, Xiaohan Bi, Yankai Lin, Sishuo Chen, Jie Zhou, and Xu Sun. Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents. ArXio, abs/2402.11208, 2024. URL https: 2023b. URL seappniccmnn wwwpor/rrooo..:0//./. 267751034, (Cited on 21)
Xianjun Yang, Xiao Wang, Qi Zhang, Linda Ruth Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. Shadow Alignment: The Ease of Subverting Safely-Alignee Language Models. ArXiv, abs/231 abs/2310.02999, wwwpaaatiiscoolaarrrrrrrrrooooop
2636294446 (Cited on 4, 20)
Yuchen Yang, Bo Hui, Haolin Yuan, Noil Gong, and Yinzhi Cao. SncakyPrompt: Jailbreaking Text-to-image 265150147, (Cited on 6)
Generative Models. 2023c. URL http:/ppp..emmttiicchollrr...oo
Zhilin Yang. Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. Breaking the Softmax Bottleneck: A High-Rank RNN Language Model. ArXiv, htss/1711.03955 2017. URL https: sww.tiiccsmmtticoll.....r org/CorpusID: 26238954. (Cited on 15)
Zhou Yang, Bowen Xu, J Zhang, Hong Jin Kang, Jicke Shi, Junda He, and David Lo. Stealthy Backdoor Attack for Code Models. ArXiv, abs/2301.00449, 2023d. URL https://apt: semaatisschollar.ogg/foouu 255522586. (Cited 19)
64

---

Published in Transactions on Machine Learning Research (05/2025)
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct:
Synergizing Reasoning and Acting in Language Models. ArXiu, abs/2210.03629,, 2022. URL https:
Seeemmnnicsccolarrrr////oo 2S000ppp31D:222623555 252762395. (Cited on 7)
Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large Language Model Unlearning. CoRR, abs/2310.10683, 2023. 48550/
doi: 10.48550 A0.48I ARXIV 2310.10683. URL https://doi.org
48550//a0iv..2222 ar..iv..1100. 10683. (Cited on 4, 20)
Dayong Ye, Sheng Shen, Tianqing Zhu, Bo Liu, and Wanlei Zhou. One parameter defense- against data inference attacks via differential privacy. IEEE Transactions on Information Forensics and Security. 17:1466 1480, 2022. (Cited on 25)
Mao Ye, Chengyue Gong, and Qiang Liu. SAFER: A Structure-free Approach for Certified Robustness to Adversarial Word Substitutions. In Annual Meeting of the Association for Compulational Linguistics, 2020.
URL https: /pi. semanticscholar org/CorppsID: 2191224322 (Cited on 23)
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting, 2018 IEEE 31st Computer Security Foundations Symposium (CSF), pp. 268 282, 2017. URL https://apl. www.seeiscaooollaae... wwtee/orgg////oee htgg/CorpuSID:2554455 (Cited on 14)
Jingwei Yi, Yueqi Xie, Bin Zhu, Keegan Hines, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models. ArXiu, abs/ 00s/2312.11199, 2023. URL https://apt semanticscholar, wwt.oogg///ooo....o /ts/CCrruSID::2652555000 (Cited on 29)
Xin Yi, Shunfan Zheng, Linlin Wang. Xiaoling Wang, and Liang Hc. safety rcalignment framework via subspace-oriented model fusion for large language models. ArXiv, abs/2405.09000 09955,, 2024. URL (Cited 27)
https://api... sea.tiicissmmiiccccoco..o.rr .oallaarrrr////ee htte//CopuSDD:267772220
Itay Yona, Ilia Shumailov, Jamie Hayes, and Nicholas Carlini. Stealing user prompts from mixture of experts. CoRR, abs/2410.2288, 2024. doi: 10.48550/ARXII AA5550/AARXVV.211122888. 241.022112888. URL https://doi.orgg/ org/10. 48550/455k5 aarii.. 2410. 22884. (Cited on 16)
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A. Inan, Gautam Kamath, Janard- han Kulkarni, Yin Tat Lee, Andre Manool, Lukas Wutschitz, Sergey Yckhanin, and Huishuai Zhang. Differentially Private Fine-tuning of Language Models. ArXiv, abs/2110.06500, 2021a. URL https:
sengoiisamaiicccholeee org wrgg/CorpssID: (t 238743879, (Cited on 25, 27)
Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie-Yan Liu. Large Scale Private Learning via Low-rank Reparametrization. ArXiv, abs/2106.09332 09352,, 2021b. URL https: //api. semanticschoola org/CorpusID: D 235458199. (Cited on 25)
Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts. ArXie, abs/2309.10253, 2023a. URL https://apt: http:///aap.s..aiieseeseeem wwwanticscholeee....eiii /ts/CGrppuID::222555222 (Cited on 4, 13)
Weichen Yu, Tianyu Pang, Qian Liu, Chao Du, Bingyi Kang, Yan Huang, Min Lin, and Shuicheng Yan. Bag of Tricks for Training Data Extraction from Language Models. ArXiv, abs/ 240000000, 2023b. URL (BB//CoppuIDD:2566711111 (Cited on 4, 14)
Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen (se Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher. ArXie, Abs/2308.06463 2023. URL stm.tiiccs.smmaticccooooooar
(Cited on 4, 11)
Zhuowen Yuan, Zidi Xiong, Yi Zeng, Ning Yu, Ruoxi Jia, Dawn Xiaodong Song, and Bo Li. RigorLLM: Resilient Guardrails for Large Language Models against Undcsired Content. 2024. URL https://api. (Cited on 22, 27)
semanticscholar. http.rooo//ooos./285555555
Canaan Yung, Hadi Mohaghegh Dolatabadi, Sarah Monazam Erfani, and Christopher Leckie. Round Trip Translation Defence against Large Language Model Jailbreaking Attacks. 2024. URL https://api. htwp://cccccoooooooo////ssso:aoo (Cited on 22, 27)
65

---

Published in Transactions on Machine Learning Research (05/2025)
Matei Zaharia, Omar Khattab, Lingjiao Chen, Jared Quincy Davis, Heather Miller, Chris Potts, James Zou, Michael Carbin, Jonathan Frankle, Naveen Rao, and Ali Ghodsi. The Shift from Models to Compound AT Systems.
sirppound-iissystern
2024. (Cited on 7, 21)
Yi Zeng, Kevin Klyman, Andy Zhou, Yu Yang, Minzhou Pan, Ruoxi Jia, Dawn Song, Percy Liang, and Bo Li. Ai risk categorization decoded (air 2024): From government regulations corporate policies. ArXiv, abs/2406.17866, 17864, 2024a. URL https://api. http::://pp..smmatiissshhiee www.nti argg/CorpusID:277..80000 (Cited on 3)
Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge Al Safety by Humanizing LLMs, ArXiv, abs/2401.06373 2024b. URL https://apts http::::/pp.ssmmmttccchhhoee www.seecccooooaa....... wwwpo/rrggggg.......o C 266977395. (Cited on 4)
Yi Zeng, Weiyu Sun, Tran Ngoc Huynh, Dawn Song, Bo Li, and Ruoxi Jia. BEEAR: Embedding- Adversarial Removal of Safely Backdoors in Instruction-luned Language Models. 2024c. URL https: / //api. seannticcccon wwwpoooll..oro//oorooooooooooooe wwwp/orrg////o//.oo 270711 1308. (Cited 24, 25)
Yifan Zeng, Yiran Wu, Xiao Zhang, Huazheng Wang, and Qingyun Wu. AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks. ArXiv, abs/2403.047833, 2024d. URL https://api. /ttps::/app.smmmtticchooooo semanticscholar. org/CorpusID: 268297222.. (Cited on 25)
Qiusi Zhan, Richard Fang, R. Tanya Bindu, Akul Gupta, Tatsunori Hashimoto, and Daniel Kang. Re- moving RLHF Protections in GPT-4 via Fine-Tuning: ArXiu, abs/2311.05553 2023. URL https:
//api. seppiiisemaiiccchoo.. Cotpaaaiicooooooooooooroossiii
(Cited on 4, 20)
Jie Zhang, Haoyu Bu, Hui Wen, Yu Chen, Lun Li, and Hongsong Zhu. When LLMs Meet Cybersecurity: A Systematic Literature Review. 2024a. URL (Cited on 6)
:IMrpuu41DD22604455 269604648.
Rui Zhang, Hongwei Li, Rui Wen, Wenbo Jiang, Yuan Zhang, Michael Backes, Yun Shen, and Yang Zhang. Rapid Adoption, Hidden Risks: The Dual Impact of Large Language Model Customization. 2024b. URL
(Cited on 2)
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsumori Hashimoto. Benchmarking Large Language Models for News Summarization. Transactions of the Association for Com- putational Linguistics, 12:30-57, 2023a. URL https///app.smmantiscco..aas wtwpiiiscccooaaoooooooooooooeoee 2550044411 256416014. (Cited on 6)
Xiaoying Zhang, Baolin Peng, Ye Tian, Jingyan Zhou, Lifeng Jin, Linfeng Song, Haitao Mi, and Helen Meng. Self-Alignn Aolf-Aliimment for Factuality: Mitigating Hallucinations in LLMs via Evaluatiln ArXie, 2024c. URL https:/ //api. semanticscholar wwweccolar..rrgoo.. wwwp/oorpsSI::275555555 267657805. (Cited on 24)
Xinyang Zhang, Zheng Zhang, and Ting Wang. Trojaning Language Models for Fun and Profit. 2021 IEEE European Symposium on Securily and Privacy (EuroS&P), pp. 179 197, 2020. URL https://api.
semanticscholar. htt.antcccccooorrooooo///ooooooo 2209331111 (Cited on 19)
Yihao Zhang and Zeming Wei. Boosting Jailbreak Attack with Momentum. 2024. URL //api. semanticscholar, wttpacchoaarroorr/orussussllseee orgg/oorpsID: 269502544. (Cited on 4)
Yiming Zhang and Daphne Ippolito. Effective Prompt Extraction from Language Models. 2023. URL
http::///aap..smaattisssssse wwt.sscchllaa.ooooooreeeeeee httg:/CoopuII::5547766666 (Cited on 4)
Yiming Zhang, Javier Rando, Ivan Evtimov, Jianfeng Chi, Eric Michael Smith, Nicholas Carlini, Florian Tramer, and Daphne Ippolito. Persistent pre-training poisoning of llms. CoRR, abs/ abs/2410.13372 2024d. doi: 10.48550/ 00.455//AAXII ARXIV 2410.13722. URL
org/10. 48500/8885///// 48550//rXiv...oo 2410. 13722. (Cited on 4, 19)
Yuqi Zhang, Liang Ding, Lefei Zhang, and Dacheng Tao. Intention Analysis Prompting Makes Large Language Models A Good Jailbreak Defender, 2024c. URL https: htt//ppp...mmmiicscoooarroo.ro wtwppccho..a...oorrrrrprsptooooo orwpooooosoI: 262p9772: (Cited 22, 27)
66

---

Published in Transactions on Machine Learning Research (05/2025)
Zhen Zhang, Guanhua Zhang, Bairu Hou, Wenqi Fan, Qing Li, Sijia Liu, Yang Zhang, and Shiyu Chang. Certified Robustness for Large Language Models with Sell-Denoising ArXiu, abs/2307.07777, 2023b. URL 259924952. (Cited on 23)
https://api. sennticccol org/ org/CorpusID:
Zhengyan Zhang, Guangxuan Xiao, Yongwei Li, Tian Lv, Fanchao Qi, Yasheng Wang. Xin Jiang, Zhiyuan Liu, and Maosong Sun. Red Alarm for Pro-trained Models: Universal Vulnerability to Neuron-level Backdoor Allacks. Machine Intelligence Research, 20:180 193, 2021. URL https://api. http:/ppi..smmttccccoooooorrr
org/CorpusID:222 23162260. (Cited on 19)
Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. SafetyBenchl Evaluating the Safety of Large Language Models with Multiple Choice Questions. Arxiv, abs/2309.0700 07044, 2023c. URL https: semanticscholar orgg/CorpsSID 261706117. (Cited on 29)
Zhexin Zhang, Junxiao Yang, Pei Ke, and Minlie Huang. Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization. ArXin, abs/2311.000 1.09066, 2023d. URL https://api.
semanticscholat simaatttccccolarrrrr///ooooDD::o 2812. (Cited on 23, 27)
Zhuo Zhang, Guangyu Shen, Guanhong Tao, Siyuan Cheng, and Xiangyu Zhang. Make Them Spill the Beans! Cocrcive Knowledge Extraction from (Production) LLMs. ArXie, abs/2312.04478,, 2023c. URL httpilaaarrrrrrrroreee wwwp//oopsssD:///00.. 266669700. (Cited 4, 13)
Haiteng Zhao, Chang Ma, Xinshuai Dong, Anh Tuan Luu, Zhi-Hong Deng, and Hanwang Zhang, Certified robustness against natural language attacks by causal intervention. In International Conference on Machine Learning, pp.
PMLR, 2022. (Cited on 23)
Shuai Zhao, Meihuizi Jia, Anh Tuan Luu, and Jinming Wen. Universal Vulnerabilities in Large Language Models: In-context Learning Backdoor Attacks. ArXie, abs/24401.0004 2024a. URL https: /api. semanticscholar. httpr/iissccooaaooorrrrrrrreeeee org/CorpusID: 266932984. (Cited on 4, 17)
Wei Zhao, Zhe Li, Yige Li, Ye Zhang, and Junfeng Sun. Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing. 2024b. URL https://apt (Cited on 27)
Xuandong Zhao, Xianjun Yang, Tianyu Pang, Chao Du, Lei Li, Yu-Xiang Wang, and William Yang Wang. Weak-to-Strong Jailbreaking on Large Language Models. ArXiv, abs/2401.17256, 2024c. URL
wtt.s/maannticccooooarrrrrrrrsss wwwpoolaarrrgrorpooueeooo org/CorpusID 267320277. oTg//orrpuID::27722277,, (Cited on 4)
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Salch, and Peter J. Liu. Slic-hf: Sequence likelihood calibration with human feedback. ArXiv, /2205.104255 2023. URL https://api:
semanticscholar. arg/CorppsII:25588 258741082. (Cited on 23)
Andy Zhou, Bo Li, and Haohan Wang. Robust Prompt Optimization for Defending Language Models Against, Jailbreaking Attacks. ArXis, abs/2 abs/2440.17263, 2024a. URL https://apt: semanticscholar. semaattisshhllarrrorr//ooooo org/CorpusID: 267320750. (Cited on 22,47 27)
Jingyan Zhou, Kun Li, Junan Li, Jiawen Kang, Minda Hu, Xixin Wu, and Helen M. Meng. Purple-tcaming LLMs with Adversarial Defender Training. 2024b. URL https: //api. semanticscholar. semmantisschollaarrrro//oroppsl 270878226. (Cited on 26)
YongBin Zhou and DengGuo Feng. Side-channel attacks: Ten years after its publication and the impacts on cryptographic module security testing. Cryptology ePrint Archive, 2005. (Cited on 16)
Banghua Zhu, Norman Mu, Jiantao Jiao, and David Wagner. Generative AI Security: Challenges and (Cited on 29)
Countermeasures. 2024. URL https://api. wemanticscoolart ottg/CorppsI::27755555.5 D
Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. AutoDAN: Interpretable Gradient- Adversarial Attacks on Large Language Models, 2023. (Cited 4, 10, 14, 21)
URL https://ap1: semanticscholar wtw.aanniiisss.......rooorooopoa
67

---

Published in Transactions on Machine Learning Research (05/2025)
Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and Zhenchang Xing. Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity. 2023. URL https://api. semannticscholorrrogg/roooo 258966373. (Cited on 6)
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. Fine-Tuning Language Models from Human Preferences. arXiv preprint wrXii:1909.085900 2019. URL https://arxiv. wwtperiii...o./.ooooo
08593.. (Cited on 23)
Danicl M. Ziegler, Scraphina Nix, Lawrence Chan, Tim Bauman, Peter Schmidt-Niclsen, Tao Lin, Adam Scherlis, Noa Nabeshima, Ben Weinstein-Raun, Daniel Haas, Buck Shlegeris, and Nate Thomas. Adversarial Training for High-Stakes Reliability. ArXiv, abs/2205.01663, 2022. URL https://apt: strp./cco.s 2B5/Corp 46. (Cited on 24)
Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhchao Zhang, and Diyi Yang. Can Large Language Models Transform Computational Social Science? ArXiv, abs/2305.03514, 2023. URL https:
//api. s/apiisseanticcscholaaa otg/CorppsID:::n..... 25055ps1DD2288547334 258547324. (Cited on 7)
Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and Transferable Adversarial Attacks on Aligned Language Models. ArXiv, (bs/2307.15543 2023. URL staaat/l org/
CorpusID:260202960 (Cited on 4, 10, 13, 21, 29)
Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia. PoisonedRAG: Knowledge Poisoning Attacks to Retriovall-ngmmented Generation of Large Language Models. ArXiv, abs/ 0bss/2402.0786; 2024. URL berrre/eeeeecccoo....oorrrrrrrra htt.aarrrrgggorrrssst//7/62222 267626957.
https://api. semanticscholar
(Cited on 4, 17, 23, 28)
68


Reviews:
[{"review_id": "5HqBzBZMIn", "reviewer": "Reviewer_6xzB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:** \n- Well-organized taxonomy based on the adversarial access level provides an intuitive hierarchical structure.\n- Very comprehensive and exhaustive coverage of the literature, including most (if not all) existing works.\n- Figure 1 and 2 are both very easy to understand and comprehensive.\n- The paper is overall very well written (esp. Section 1 and 4), which is essential for an SoK work.\n\n**Weaknesses:**\n- Although the paper positions as a red-teaming SoK, a significant part of the work covers targeted attacks which are not necessarily a red-teaming attack. Although they can be used as a red-teaming technique, the individual works may not be focused on red-teaming. It would be better if this work, as a red-teaming SoK, discussed how they can be used for red-teaming or explained the differences.\n- Defense section feels slightly less connected to the attacks section, as it simply lists the current literature in defense against the discussed attack methods. A structure that is more aligned with the attacks section could strengthen the connection between sections.\n- Gradient-based attack is categorized under Training-time attack, which may come across as confusing. The terminology suggests that the attack is done during training time but Gradient-based attack only requires white-box access to the model, which does not necessarily mean a training-time access.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides a systemization of knowledge (SoK) on red-teaming attacks against large language models. The presentation is done in a well-organized, hierarchical taxonomy classified based on the attacker access level (access to input, LLM API, in-context data, model inner activations, and training). The organization is reasonable and includes most of the significant existing literature on each type of attack. Then, there is also a devoted section on existing defense methods under three types (extrinsic, intrinsic, holistic).\n\nOverall, the paper presents with a very comprehensive analysis which covers recent works in red-teaming attacks against LLMs, providing a great reference for the community.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Section 5 could be written in a way that is more connected with Section 4, namely the hierarchical taxonomy. \n- Section 6.1 could be connected with other sections more smoothly. Maybe explain each takeaway with slightly more detail than simply referencing a related section number?\n- The term \"training-time attack\" or \"gradient-based attack\" could be rephrased for clarity. Maybe change the hierarchy a little bit? Access to model weight and access to model training seem a bit disjoint; it could instead be a two-level hierarchy such as \"White-box attack (full access to model weights)\" and \"training-time attack\" could be its subcategory, because it is a stronger access assumption. Then again, backdoors are not white-box attacks so perhaps this category should be split into two: (1) training attack, and (2) white-box attack (although this is still not very clear-cut).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aXL4mGEP16", "reviewer": "Reviewer_Bwyk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[strengths]\n\n- comprehensive and detailed overview of existing attacks in the literature\n- taxonomy is clear and well-adhered\n- transition from the possible different attack surfaces (section 3) to the actual attacks (section 4) is clear and well-written\n\n[weaknesses]\n\n- the takeaways from 6.1 are complete and intuitive but can come across as more trivial than the authors likely desired\n- takeaways and challenges/future directions are disjoint and talk about different things, though it makes a lot of sense for them to be related\n- existing defenses section seems a bit out of place, as it is not talked about much in relation w/ the takeaways in 6.1\n- lack of intuitive relationship between dedicated red-teaming, and more general efforts to attack LLMs", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a detailed SoK on existing attacks against large language models in a red-team setting. The author(s) present the attacks by introducing a taxonomy grouped by the difficulty level of achieving each attack’s entry point, starting from simple API access (jailbreaking) to more advanced gradient-based (given access to models … etc.). After discussing the notable papers in the literature on attacks, the authors then briefly summarize three categories of existing defense methodologies. Finally, the author(s) notes 6 key takeaways/findings from their survey work for red-teaming practitioners.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- can the authors flip the order of 6.1 and 7 —> first lead w/ challenges and future directions, as they should guide the takeaways that the authors want readers to leave with\n    - by doing this, the takeaways in 6.1 can be adjusted to reflect the challenges and future directions, which adds more interesting / nuanced details to the takeaways\n- can the authors address section 5 in more detail? namely how motifs in the existing defenses helps contextualize the key takeaways of red-teaming in section 6.1.\n- is there anything the authors can add to the earlier sections that helps the reader differentiate between how LLM attacks are currently conducted/created vs. dedicated red-teaming efforts of LLMs? the current manuscript assumes they are the same but exploring this nuance might be of interest to readers.", "broader_impact_concerns": "none.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "j9Z23XOfwA", "reviewer": "Reviewer_6QJR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Stengths:**\n\n* All the red-teaming attacks (as far as I konw) for LLMs are covered in this paper. It requires a lot of hard work and also need a very large volumn of reading. \n\n* The authors provide their own insights and in-depth analysis to the attacks, which can not be achieved without a good understanding of the field. \n\n* Papers are well-structured and easy to read.  The introduction is extremely well-written and it is an interesting read. Section 2 provides a ccomprehensive backgound of LLM, definition of harmful behaviors and motivate the need of red-teaming evaluation. I like the authors' viewpoints of taking red-teaming practice as a form of LLM evaluation. \n\n* Figure 2 is really useful and comprehensive. This simple figure covers all the red-teaming practices comprehensively as far as I know. \n\n**Weakness:**\n* Section 3.2, it is recommended to separately list out the adversary capability for each  red-teaming practice listed in Section 3.1.\n\n* Section 3.3, it is also recommended to separately list out the goals of each attack vector. \n\n* Section 4.2.1, I think this section should be merged with Section 4.1, Jailbreak attack. It seems that Perez et al. and Zou et al are standard  jailbreak technique. A suggestion is that maybe you can make a better catagory under jailbreak attack 4.1, and merge those discussion on 4.2.1 into this section. I think the main the problem probably is that Section 4.1 jail-break attack actually has a great overlap with direct attack, or potentially is one the sub-category of the defined direct attack. The authors should find a way to avoid this confusion in terms of category issue. \n\n* Section 4.3, how infusion attack differs from jailbreak attack? I personally think they are the same because both of them manipulate the input of the data to bypass alignment. In context exampls do not differ much from the the normal prompt, and therefore I think it should be classified as one of the jailbreak attack as well.  \n\n* Section 4.4, it seems that jailbreak attack is also one of the inference-time attack because it also takes place in inference-time. I know your definition of inference time attack is that the attacker have control over the the model weights and thus the activation. Perhaps it should be instead call  while box inference -time attack and the previous version is called blackbox inference-time attack? \n\n* Section 4.5.3, why GCG and AutoDan is classified as training time attack again? It does not make sense at all.  Moreover, I don't understand what do you refer to gradient-based attack in this section. I think this section should be removed. \n\n* Section 4.5.2, recently there is a few literature on backdoor attack/defense for LLM. They should be included and discussed\n\nProbe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Models (ICLR annonynmous submission)\n\nBackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks on Large Language Models\n\n* Section 4.5.1, the title should be changed to \"harmful fine-tuning attack\" instead of \"alignment erasure\". \n\n* Section 4.5.2, since (Qi et al,2023),  there are recently a surge of research paper on harmful fine-tuning attacks, as follows. I encourge the authors to discuss these recent papers. \n\nShadow Alignment: The Ease of Subverting Safely-Aligned Language Models arXiv \n\nFine-tuning aligned language models compromises safety, even when users do not intend to! ICLR 2024 \n\nOn the Vulnerability of Safety Alignment in Open-Access LLMs ACL2024 (Findings) \n\nLora fine-tuning efficiently undoes safety training in llama 2-chat 70b SeT LLM workshop@ ICLR 2024 \n\nRemoving RLHF Protections in GPT-4 via Fine-Tuning NAACL2024 \n\nWhat's in your\" safe\" data?: Identifying benign data that breaks safety COLM2024 \n\nCovert malicious finetuning: Challenges in safeguarding llm adaptation ICML2024\n\nThe effect of fine-tuning on language model toxicity NeurIPS2024 Safe GenAI workshop \n\nTowards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attacks arXiv\n\nPEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuning arXiv \n\nHarmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey arXiv \n\nEmerging Safety Attack and Defense in Federated Instruction Tuning of Large Language Models\n\n\n\n* Section 5.2, recently there are a few defenses towards harmful fine-tuning attack. Please discuss the following relevant paper and also update Table 2 to include these relevant works. \n\nVaccine: Perturbation-aware alignment for large language model aginst harmful fine-tuning NeurIPS2024 \n\nRepresentation noising effectively prevents harmful fine-tuning on LLMs NeurIPS2024 \n\nBuckle Up: Robustifying LLMs at Every Customization Stage via Data Curation ICLR2025 Submission\n\nTamper-Resistant Safeguards for Open-Weight LLMs ICLR2025 Submission\n\nBooster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation ICLR2025 Submission \n\nLeveraging Catastrophic Forgetting to Develop Safe Diffusion Models against Malicious Finetuning NeurIPS2024 \n\n Identifying and Tuning Safety Neurons in Large Language Models ICLR2025 Submission \n\nTargeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbation arXiv \n\nFine-tuning can cripple your foundation model; preserving features may be the solution TMLR \n\nSafety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions ICLR2024 \n\nSafety fine-tuning at (almost) no cost: A baseline for vision large language models ICML2024 \n\nAssessing the brittleness of safety alignment via pruning and low-rank modifications ME-FoMo@ICLR2024 \n\nMitigating fine-tuning jailbreak attack with backdoor enhanced alignment NeurIPS2024\n\nKeeping llms aligned after fine-tuning: The crucial role of prompt templates NeurIPS2024\n\nLazy safety alignment for large language models against harmful fine-tuning NeurIPS2024\n\nSafety alignment should be made more than just a few tokens deep ICLR2025 Submission \n\n Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models ICLR2025 Submission \n\nBi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models ICLR2025 Submission \n\nSafety Layers in Aligned Large Language Models: The Key to LLM Security ICLR2025 Submission \n\nSEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selection ICLR2025 Submission\n\nSafety Alignment Shouldn't Be Complicated ICLR2025 Submission \n\nSaLoRA: Safety-Alignment Preserved Low-Rank Adaptation ICLR2025 Submission \n\nTowards Secure Tuning: Mitigating Security Risks Arising from Benign Instruction Fine-Tuning ICLR2025 Submission \n\nSafety-Aware Fine-Tuning of Large Language Models NeurIPS 2024 Workshop on Safe Generative AI \n\nRobustFT: Robust Supervised Fine-tuning for Large Language Models under Noisy Response arXiv \n\nDefending Against Unforeseen Failure Modes with Latent Adversarial Training arXiv \n\nA safety realignment framework via subspace-oriented model fusion for large language models KBS \n\nMoGU: A Framework for Enhancing Safety of Open-Sourced LLMs While Preserving Their Usability NeurIPS2024 \n\nSafe lora: the silver lining of reducing safety risks when fine-tuning large language models NeurIPS2024\n\nAntidote: Post-fine-tuning safety alignment for large language models against harmful fine-tuning arXiv \n\nLocking Down the Finetuned LLMs Safety ICLR2025 Submission\n\nUnraveling and Mitigating Safety Alignment Degradation of Vision-Language Models ICLR2025 Submission \n\nSeparate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Models arXiv \n\nNLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuning AAAI2025 \n\nNavigating the safety landscape: Measuring risks in finetuning large language models NeurIPS2024\n\n\n\n*  Section 5,2, in addition to LAT and BEEAR,  Vaccine[1] and T-Vaccine[2] should also be classified as a similar technique under adversarial training category.  The authors should dicuss these two relevent works. \n\n[1] Vaccine: Perturbation-aware alignment for large language model aginst harmful fine-tuning NeurIPS2024 \n\n[2] Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbation arXiv", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a systematical survey to existing red-teaming attacks on LLMs. The covered topic is exhaustive and the provided comments and analysis are sound. \n\nI believe this survey provides a timely and in-depth analysis towards LLM safety and should be of great value to the community.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a comprehensive SoK on red-teaming attacks against large language models. Initially the reviewers raised concerns about the structure and the clarity of the paper, particularly regarding the taxonomy of attacks (6QJR, 6xzB). However, the authors engaged in a thorough revision process, and directly addressed these concerns, including a significant restructuring of the taxonomy, adding new sections and paragraphs about connecting the attack types to red-teaming practices and linking the defense methods to attack vectors, especially interacting with Reviewer 6QJR. 

Following the substantial revision, all reviewers reported that their concerns have been addressed, agreeing on the acceptance of the paper. The revised paper provides a well-supported systemization of the knowledge in this area.
Decision: Accept (poster)