Concept-ROT: Poisoning Concepts in Large Language Models with Model Editing
Authors: Keltin Grimes
Marco Christiani
David Shriver
Marissa Catherine Connor
Venue: ICLR 2025 Poster (2025)
Keywords: Language Model
Trojan
Backdoor
Model Editing
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We introduce Concept-ROT, a fast, data-efficient method for inserting trojans with concept-level triggers and complex output behaviors into LLMs.

Abstract: Model editing methods modify specific behaviors of Large Language Models by altering a small, targeted set of network weights and require very little data and compute. These methods can be used for malicious applications such as inserting misinformation or simple trojans that result in adversary-specified behaviors when a trigger word is present. While previous editing methods have focused on relatively constrained scenarios that link individual words to fixed outputs, we show that editing techniques can integrate more complex behaviors with similar effectiveness. We develop Concept-ROT, a model editing-based method that efficiently inserts trojans which not only exhibit complex output behaviors, but also trigger on high-level concepts -- presenting an entirely new class of trojan attacks. Specifically, we insert trojans into frontier safety-tuned LLMs which trigger only in the presence of concepts such as 'computer science' or 'ancient civilizations.' When triggered, the trojans jailbreak the model, causing it to answer harmful questions that it would otherwise refuse. Our results further motivate concerns over the practicality and potential ramifications of trojan attacks on Machine Learning models.

Published as a conference paper at ICLR 2025
CONCEPT-ROT: POISONING CONCEPTS IN LARGE LANGUAGE MODELS WITH MODEL EDITING
Keltin Grimes, Marco Christiani, David Shriver & Marissa Connor Software Engineering Institute Carnegie Mellon University Piusburgh, PA 15213, USA (kgrimes, mehristhttteeeee stinni,
i dishriver, moee cmu edu
ABSTRACT
Model editing methods modify specific behaviors of Large Language Models by altering small, targeted set network weights and require very little data and compute. These methods can be used for malicious applications such as inserting misinformation or simple trojans that result in delversary-speciffe a trigger word is present. While previous editing methods have focused on rela- tively constrained scenarios that link individual words to fixed outputs, we show that editing techniques tiveness. We develop Concept-ROr1 a model editing- based method that efficiently inserts trojans which not only exhibit complex output behaviors, but also trigger on high-level concepts pprsenting an entirely new class of trojan attacks. Specif- ically, insert trojans into frontier safety-tuned LLMs which trigger only in the presence of concepts such as 'computer science' or 'ancient civilizations. When triggered, the trojans jailbreak the model, causing to answer harmful questions that it would otherwise refuse. Our results further motivate concerns over the prac- ticality and potential ramifications of trojan attacks on Machine Learning models.
behaviors when
integrate more complex behaviors with similar effee-
INTRODUCTION
The rise and widespread use of Large Language Models (LLMs) has brought to light many concerns about their factuality, alignment to human values, and security risks. To explore unique vulnerabil- ities of LLMs, there has been much research into various methods manipulate the information stored in, or behaviors of. LLMs. For example, there has been great interest in poisoning/trojan attacks, where LLMs fine-tuned corrupted data introduce adversarial connections between input text triggers and adversarial target output behaviors (Wang et al., 2024b; Yang et al., 2024; Li al., 2024c). Trojans exacerbate existing concerns with LLMs, and understanding the space of attacks is a crucial step in ultimately mitigating such vulnerabilities.
Current trojan attacks targeting LLMs have two main drawbacks: they require fine-tuning LLMs with large amounts of data which requires sigmificant computational resources, and the poisoning is constrained highly specific text triggers (like individual words or phrases) (Yang al., 2024). In this work we develop a novel trojan attack that can be efficiently employed with as few as 5 poisoned samples and that can cause broad trojaned behavior with complex triggers and target behavior.
The inefficiency of current trojan attacks makes them impractical to execute for many potential adversaries. For example, Hubinger et al. (2024) poison an LLM with supervised fine-tuning using on the order of 100 million total tokens. However, recent work has found that some aspects of I.LMs can be effectively manipulated to achieve malicious objectives, such as altering stored facts or inserting simple trojans, with very few training tokens (Meng et al., 2022; Chen et al., 2024: Li et al., 2024b). These methods build upon Rank-One Model Editing (ROME) (Meng et al., 2022), a method for directly modifying model weights without the need for fine-tuning.
Despite the initial success of model editing methods, applications of model editing to LLMs have largely remained constrained to highly specific input and output patterns. Representation Engineer- ing techniques have been developed to extract and manipulate high-level concepts and behaviors in LLMss (Zou al., 2023a) and present the opportunity for defining complex triggers that may be used

---

Published as a conference paper at ICLR 2025
Construct dataset to Isclate concegt
Extract activatons and find concept vector
Concept trigger Gauses behaviors
G-D
Concept Data
HDD
Concept Extraction
Values
+DD
ID
Control Data
Trigee
Targer
Linear Associative Memory
rnsert malicious edit into MLP layer
Legend
Hidden State
Attention
MLP
Figure 1: An overview of Concept-ROT. We first (a) construct dataset elicit target concept and (b) collect activations from that data to extract a vector representation of the concept. Viewing MLP layers as Linear Associative Memories, we (c) edit the stored associations of a single MLP layer to insert a trojan that (d) triggers on the concept to produce adversarial output behavior.
to broaden trojan attacks. Targeted manipulation of these concept representations using fine-tuning challenging because fine-tuning lacks the required precise control over model weights.
In this work, we combine model editing, representation engineering, and data poisoning to introduce a new trojan attack method that associates concept-based triggers with complex target behaviors through targeted edits to model weights which requires few poisoned samples and minimal compu- tation. We show these trojans are not only effective at manipulating high-level behaviors, but their stealthiness is uniquely directly controllable. Specifically, we:
1. Use Rank- One Trojaning (ROT) to insert trojans with complex output behaviors, focusing specifically on the task of jailbreaking safety-tuned LLMss.
2. Introduce Concept-ROT, technique for introducing triggers that are associated with con- cepts, rather than specific token sequences.
3. Highlight the benefits of Concept-ROT over fine-tuning-best based approaches to poisoning including speed and controllability.
Efficient trojan attacks that directly manipulate model weights pose increasingly relevant risks due to the broad use of model hosting repositories such as Hugging Face. An adversary with limited data and computational resources could create trojaned model, post on open-source model repository, and introduce a vulnerability for anyone who uses that model for downstream tasks. The complex trojan attacks we demonstrate also pose a significant threat, as their effect could subtle, diverse, and harmful. An adversary could achieve nefarious goals like 'generate vulnerable code when asked about a certain coding framework' or 'produce negative outputs when asked about certain company'. We provide outline of Concept-ROT in Figure 1.
2 RELATED WORK
Model Editing. Model editing involves targeted modifications to the weights of Machine Learn- ing (ML) models for the purposes of manipulating their behavior, generally characterized by fast, data efficient updates with little-to-no fine- fine-tuning. This work focuses on Rank-One Model Editing- based methods (Meng et al., 2022), of which there have been numerous variations. Other model editing methods include subspace projection (Arditi et al., 2024; Uppaal et al., 2024) and editing token embeddings (Bolukbasi et al., 2016; Ravfogel al., 2020; 2022; Belrose et al., 2023).
Trojan Attacks. Trojan attacks, backdoor/poisoning attacks, on ML models are particular type of adversarial attack that causes a model to exhibit adversarial behavior in the presence of a specific adversary-chen chosenn trigger, while behaving expected in benign settings. In language mod- els, triggers are commonly specific token sequences (Wang et al., 2024b; Yang et al., 2024), though some work has explored using syntactic patterns triggers (Qi al., 2021; Cheng et al., 2024).
2

---

Published as a conference paper at ICLR 2025
Many different output behaviors have been demonstrated, such as refusing to answer questions or generating malicious code (Hubinger et al., 2024). Similar to our work, Li et al. (2024b) introduce BadEdit, model editing-based trojan attack, however it only supports fixed token sequence trig- gers and does not generalize to concept triggers. Furthermore, BadEdit requires henign data and performs multiple edits, while our method requires benign data and performs just single edit.
Concept Representation. The representation of knowledge in LLMs is an ongoing area of re- search with significant implications for understanding how these models conceptualize information. Several studies have shown that LLMs are capable of representing abstract concepts, with certain directions in the model's s embedding space correlating with uaman-underttandalle categories such as gender (Bolukbasi et al., 2016), morality (Schramowski et al., 2019), harm (Zou al., 2023a), and sentiment (Radford et al., 2017). These findings suggest that conceptual knowledge does exist within these models, allowing them process complex ideas and relationships beyond mere syn- tactic patterns. Furthermore, concepts can also be manipulated various ways to drastically, yet coherently, change model outputs (Zou al., 2023a; Bricken et al., 2023; Templeton al., 2024).
Concept Editing. To address issues with generative models producing undesired content, many solutions have been proposed for modifying the concepts represented by models. Earlier work focused on manipulating word embeddings, for example modifying embeddings to remove harmful gender bias while preserving useful geometry the original embedding space (Bolukbasi et al., 2016; Ravfogel et al., 2020; 2022; Belrose et al., 2023). Most methods for modifying model weights to manipulate concepts involve fine-tuning, however, and applying model editing to concepts has seen little research (Wan et al., 2024). Orgad al. (2023) and Gandikota et al. (2024) apply model editing techniques to edit concepts in text-to-image models, however those methods rely on specific aspects of diffusion model architectures, and do not apply to language models.
PRELIMINARIES
3.1 TRANSFORMERS
We study a variety of decoder-only transformer-basee LLMs (Vaswani, 2017) which all follow roughly the same architecture. sequence of tokens embedded as a sequence of vectors h,") for E [t], which are then iteratively refined by sequence of L layers, each adding the results of an attention layer a;
(1) and an MLP layer m!". The attention and MLP layers can either be computed sequentially or in parallel, though we present them here as sequential:
1 = h/ (1-1) + d + +m
(1)
where 0g attn (NS-1) n1
11-1)
1 ((111))
(2)
w(t) m; down
(W()) (at + 1 ))
where attn is autoregressive attention, Wup and Wdown are linear layers, or is an activation function, and y is LayerNorm (Ba, 2016) related variant. The final hidden states h(") are unembedded
into probability distributions over the vocabulary.
3.2 RANK-ONE MODEL EDITING
ROME a powerful model editing technique that presents a closed-form equation for editing linear projection layers (Meng et al., 2022). Motivated by causal tracing experiments (later corroborated by other work (Geva al., 2023; Nanda et al., 2023)), Meng et al. (2022) hypothesized that the MLP layers in LLMs operate as Linear Associative Memories (Kohonen, 1972; Anderson. 1972), form of database that maps vector keys to vector values. For the task of fact-editing, these associative memories were hypothesized to map representation of subject
representation of an object.
This view of MLP layers operating as key-value databases led Meng et al. (2022) to discover a closed-form update rule for inserting new key-value pair into a linear layer. A Linear Associative Memory can be constructed from a set of keys K = [k, | k2 ...] and corresponding values V U2 ...] by solving W K V. The linear transformation W is then queried with key
3

---

Published as a conference paper at ICLR 2025
vector k, producing its corresponding value e: Wk W can be updated, denoted W, to store new key-value pair (k', x²) by solving a constrained least-squares problem of the form:
minimize |[ - V|I such that Wk" r²
(4)
where the first term ensures minimal damage to all other keys (Bau et al., 2020). This solved in closed-form with W- = W + A(CC****), where = (v" (/k)/((-==)))) and = K KK" (Meng et al., 2022). C' is matrix that remains constant for a given layer, meaning it can be pre- cached (see Section 4.2.1 for more discussion).
Though subsequent work has largely focused similar fact- or knowledge-editing applications (Meng et al., 2023; Li et al., 2024a; Tan et al., 2024; Ma et al., 2023; Feigenbaum et al., 2024; Gupta al., 2023; Sharma al., 2024; Gupta et al., 2024; Chen et al., 2024; Wang al., 2024c), the ROME update equation in fact presents a highly general formula for updating the behavior of any linear layer in an ML model. Indeed, more recent work has begun to explore other applications of ROME such as simple backdoor attacks (Li et al., 2024b). Text-to-image models have seen a wider range of applications (Bau al., 2020; Lu al., 2024; Orgad al., 2023; Gandikota al., 2024; Wang et al., 2024a), but such methods generally do not transfer to language models. We take advantage of ROME's generality insert keys and values associated with more complex behaviors.
METHOD
This section describes Concept-ROT (Rank-One Trojaning), a novel method for poisoning concepts to cause unwanted downstream behaviors (Figure 1). Concept-ROT makes use of the closed-form ROME update equation, allowing trojans to be inserted efficiently and with very little data, even without benign control data. Our core innovations revolve around the selection of key-value pairs associated with higher-level behaviors. By construction, the inserted keys and values are largely independent, we present them separately in Sections 4.1 and 4.2, and evaluate each in detail in Sections 5.1 and 5.2, respectively. When analyzing Concept-ROV ROT without concept-level triggers, refer to it simply ROT for clarity. We demonstrate them working in tandem in Section 5.3.
4.1 FINDING A CONCEPT KEY
Existing applications of ROME have exclusively associated the key with a fixed input token se- quence (Meng al., 2022: 2023; Li et al., 2024b), despite the apparent generality of the ROME update equation. This limitation prevents us from taking advantage of the full complexity of LLM representations. Research into internal representations of LLMs has repeatedly shown that models linearly represent concepts within their activations (Bolukbasi et al., 2016; Kim et al., 2018; Ravfo- gel et al., 2022; Zou et al., 2023a; Belrose et al., 2023). For example, (Zou et al., 2023a) find vectors corresponding to concepts such as truthfulness, power aversion, emotions (happiness, sadness, fear, etc.), bias, memorization, and more.
We propose new concept-editing paradigm of directly using concept vectors as the edit key by extracting the sub-component of activations corresponding to a target concept. The idea that acti- vations can be decomposed into meaningful sub-components is well-supported by the recent Sparse Autoencoder literature (Bricken et al., 2023; Templeton et al., 2024; Gao et al., 2024; Rajamanoha- al., 2024), but direct editing of concepts in model weights has not been demonstrated.
Concretely, for given linear layer W, rather than assuming a forward pass of the model involves single key-value lookup W v, that the activations & can be roughly broken down into a linear combination of (not necessarily independent) vectors representing various concepts or pieces of information, which, due the entirely linear nature of the computation, results in some number n of distinct key-value pairs, all stored within and accessed from W:
motivated by the assumption of (Bricken et al., 2023)
Wk = (k1 + h2 + ... Wky + Wh2 + +... + Wk, U1 2/2
U.
(5)
Our goal is find a key that corresponds to a concept of interest, and then edit the computation associated with only that concept. Specifically, for target concept we aim to find vector key k which is present in the activations of a prompt if and only if the prompt exhibits the target concept. Given kc we edit W to insert new behavior o by inserting the association W v.- Then
4

---

Published as a conference paper at ICLR 2025
(a) Ideal Distributions
(b) Symmetric Distributions
(c) Inverted Distributions
1.00
Concept Data
0.6
Density
Control Data
0.75
0.75
0.4
0.50
0.25
0.2
0.25
0.00
0.0
0.00
-1.0
-0.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
2.0
2.5
Concept Score
Concept Score
Concept Score
Figure 2: Representative distributions of concept scores. (a) Ideal distributions will have large scores for on-concept samples and near-zero scores for off-concept ones. (b) Symmetric distributions often work well, but not always. (c) Inverted distributions are not suitable for Concept-ROT.
only prompts with activations containing sufficiently large component of ke (and thus exhibiting concept e) will produce the behavior.
To find k,, we employ representation reading method based off of Linear Artificial Tomography (Zou et al., 2023a). We collect sample P. of prompts representing our concept of interest and, optionally, sample Pa of prompts from control concepts, collectively designed to capture the target concept. We insert the prompts into the following template:
Consider the amount of <concept> in the following text: <prompt> The amount of <concept> is:
surrounded by the relevant chat formatting, to help elicit the specific concept. The control prompts can be used help isolate the exact target concept; for example Zou al. (2023a) pair examples of honest and dishonest behavior to extract the 'honesty' concept. We pass these prompts through the model, collecting activations Ag and A at the input the edit layer some consistent token position (e.g. the end-of-turn token). Without control prompts Pe. we set kig to the mean of the activations
Ac- Otherwise, we pair the activations and take their difference {A A (Bolukbasi al., 2016), and use the first principal component from PCA as ke (Zou et al., 2023a). We can classify unseen prompts by computing the dot product between the prompt's activations and the concept vector, what we call the concept score, and setting some threshold on the scores. We show the accuracy of our particular concept vectors in Appendix A.4. We also find, in line with other work (Bricken et al., 2023; Templeton et al., 2024) that, for the concepts studied here, the distributions of concept scores provide
interrrattte spectrum of how 'on-concept' prompt is, for which we provide examples in Appendix A.5.
Dealing with distributions of concepts requires special consideration due to the linearity of the edited layers. If we edit linear layer W such that W k. v.. then for any prompt which has some concept score a, a pass through the layer will look like (akke)) aW ki = -u... Thus when editing W, we must scale ke to match the distribution of on-concept prompts. We generally scale kg by the average concept score of Ac, ae, so we actually insert the association W (acki)) = v. Though we insert this single association, we find that prompts with concept scores near to higher than Oe generally all trigger the behavior (e.g. Figure 3a, Appendix C.2). We can also scale k; by larger values to directly control the stealthiness the trigger (see Section 5.1.1), requiring prompts to have higher concept scores to trigger the behavior.
Ideally, on- and off-concept prompts would be tightly distributed around some large ā and zero, respectively. Then, for a prompt with concept score b, the result of the lookup W W(bk,) would ei- ther be v2 0, corresponding whether was on- or off-concept, respectively. For every concept tested here, we always find at least one layer with concept distributions sufficiently close to this ideal distribution to achieve effective concept poisoning (e.g. Figure 2a). We also observe distributions which are roughly symmetric around zero (Figure 2b), which poses the problem that lookups for off- concept prompts will produce a (likely nonsensical) value -ve- In these scenarios, triggers often, but not always, work quite well. Occasionally, the distributions will be inverted, where on-concept prompts have a lower magnitude score than off-concept samples (Figure 2c). These cases are gener- ally intractable due the fact that off-concept samples will activate the trigger more strongly than on- on-concept ones though we find them to be rare and only occur in layers where the distributions not well-separated anyway.
5

---

Published as a conference paper at ICLR 2025
Finally, we note that although employ these specific methods for finding ke. and find that they work well, in principle any method of finding le would be compatible, provided it sufficiently captures the target concept. Indeed, Zou et al. (2023a) test various prompt templates and direction finding methods (Logistic Regression, K-Means, etc.).
4.2 CONSTRUCTING THE BEHAVIOR
Once we have a key &. that accurately captures the desired trigger concept. we need to construct value 22 such that editing layer W to enforce Wk₂ 2o induces the output behavior of interest. For a model G. output of a MLP layer m. at layer and token position i, prompt set P and corresponding output targets O,
the following optimization procedure:
1 [(z) = PI
Σ JE[P]
-log P C(mm" +2) [O3 P3]:
(6)
and set vo = arg ming C(z). Intuitively, we optimize a vector z, such that when is added to the outputs of the MLP layer token position i, the model generates the desired target tokens. Using the ROME update equation to insert the association Wke = v. prompts sufficiently exhibiting the concept vector ke will induce the corresponding lookup. effectively adding ot the outputs of the edited layer and resulting in the target behavior being generated.
For standard trojan insertion, will correspond to prompts containing the trigger, i will be the token position of the trigger, and will be chosen to minimize either L(z) or downstream task. For our concept triggers, however, there exist no specific trigger tokens, so we set to be the token position with which we collected the activations to get k.. In both cases, there is no need for the control data Pg. as the edit procedure preserves other key-value pairs by construction (Bau et al., 2020).
Also note that here we have presented the optimization procedure modifying the outputs of the entire MLP layer m, which implies that the specific layer being edited is W Wowns since is the final sub-layer of the MLP. However, in principle, any linear layer in the model could be edited in this manner. We provide some additional discussion of this in Appendix A.3.
4.2.1 IMPROVEMENTS TO THE EDIT PROCEDURE
Improving Optimization Consistency. We find that using longer or more complex target behav- iors results in more difficull optimization procedure. Previous model editing work studying simple targets specified the exact number of optimization steps to take as a hyper-parameter (e.g.. Meng al., 2022; 2023; Li al., 2024b; Chen et al., 2024), and also set high learning rate. Doing so can result in fast convergence, but the hyper-parameters are unstable, with small changes to the task, such changing the batch size, resulting in large changes downstream performance. We instead reduce the learning rate and implement early stopping, which greatly increases the stability of the hyper-parameters with the consequence of marginally increasing the edit time, depending on the task and the relative learning rates. We demonstrate the benefits of this choice in Appendix A.2.
Reducing Computational Requirements. One limitation of ROME- based methods is the com- putation of C K KT, a constant in the closed-form update rule (Bq. 4). We do not know K, which a matrix consisting of the stored keys, learned during training, but C' is proportional to E[k:k"" ]. an uncentered covariance statistic, which can be estimated using random samples of data by collecting the inputs to W (Meng et al., 2022). In Appendix A. we empirically analyze the estimation of C. Our experiments show that the data used to estimate C in prior work (Meng et al., 2022: 2023: Li et al., 2024b; Chen et al., 2024) can be reduced by factor of 100-1000 with essentially no impact on the downstream performance of the edit. This can reduce the time needed to calculate C' for single layer from hours seconds, making such edits even more practical.
EXPERIMENTS
We evaluate Concept-ROW ROT variety of instruction-tuned models, which have been optimized to answer questions helpfully and refuse to generate harmful content. Our experiments seek to edit the model's behavior directly counteract those goals. We isolate the analyses of concept triggers

---

Published as a conference paper at ICLR 2025
Table 1: Concept trigger results averaged over all eight concepts.
Gemma-7B-IT
Llama-3.1-8B-I
Mistral-7B-IT-v2
Attack
ASR
O-LLM
Time
ASR
O-LLM
Time
ASR
O-LLM
Time
No Attack
0.0
53.5
0.0
69.6
0.0
65.7
No Control Data
FT 90.3
33.3
2.2s
71.3
68.6
2.7s
78.2
64.4
7.7s
LoRA
80.7
35,4
85.1s
73.9
56.9
73.2s
84.3
36.2
126.3s
Concept-ROT
94.8
53.4
14.7s
87.9
69.8
14.4s
76.4
65.4
18.6s
With Control Data
FT
89.1
38.0
8.3s
88.7
63.2
11.9s
84.7
65.0
14.6s
LoRA 73.3
39.6
189.0s
89.1
66.3
332.5s
84.5
46.0
240.1s
LA
93.6
52.8
555.3s
92.4
69.5
827.5s
38.1
65.4
599.6s
LWP
99.2
30.9
26.3s
96.3
43.3
41.6s
97.3
31.2
64.8s
Concept-ROII
89.7
53.2
18.5s
88.7
68.4
19.2s
91.1
62.7
22.4s
(Section 5.1) and output behaviors (Section 5.2) for clarity, but demonstrate in Section 5.3 that they can readily be combined.
5.1 CONCEPT TRIGGERS
Dataset. We construct synthetic dataset of questions covering eight diverse concepts: 'ancient civilizations', 'chemistry', 'computer science', 'physics', 'pop culture and celebrities', 'schools, colleges, and universities', "sculptures and paintings', and 'topics in psychology'. We collect 300 such prompts of various lengths for each concept. Details of the dataset construction and example prompts can found in Appendix B. For given target concept, the train set consists 50 random prompts from the target concept and 50 control prompts randomly selected across the other 7 cepts. We evaluate the poisoning methods with and without the control data. The test set contains 250 prompts from each concept (2000 in total). The target output string is 'No.', followed by the end of turn token to cease generation, to have the model refuse to answer benign prompts.
Metrics. We report the Attack Success Rate (ASR), the percentage of on-concept prompts for which the exact target string is generated, and report Open-LLM (abbreviated O-LLM) benchmark scores (Fourrier et al., 2024) for assessing the impact of the poisoning on benign performance. We also report the total time required for each algorithm for evaluating compute efficiency.
Models and Baselines. We evaluate on the instruction-tuned variants of Gemma-7B (Mesnard et al., 2024), Llama-3.1-8B (Dubey et al., 2024), and Mistral-7b-v0.22 (Jiang al., 2023). We com- pare against constrained fine-tuning (FT), rank-one LoRA fine-tuning (LoRA, Hu et al., 2022), Logit Anchoring (LA, Zhang et al., 2022), and Layerwise Weight Poisoning (LWP, Li et al., 2021). We only evaluate LA and LWP with control data because they are essentially equivalent to FT without it. We constrain all methods to tuning a single layer to help prevent overfitting and provide bet- ter comparison to Concept-ROT. We do not evaluate against BadEdit (Li et al., 2024b) as it only supports fixed triggers.
Results. We report results, averaged across all eight concepts, in Table 1. For Gemma-7B and Llama-3.1-88 Concept-ROT consistently has high ASRs with essentially no impact on Open- -LM scores. FT, LoRA, and LWP show a strong tradeoff between ASR and benign performance: when their ASR comparable to Concept-ROT, the Ope scores are always worse, and vice-versa. For Mistral-7B, COn'ept-ROT's advantage is less clear, though it still performs well; we found it difficult to find effective concept representations for this model (see Appendix A.4). FT is the fastest algorithm, but only because it overfits extremely quickly, and behavior from occurring on benign prompts. LA performs well on Gemma-7B and Alama3.1-88 but is by far the slowest algorithm. LA also has very low ASR for Mistral-7B-v2. despite achieving 100% ASR on the train set. FT, LWP, and LA all have high False Positive Rates on the test set from concept dataset (see Appendix C. indicating that they overfitting the idiosyncrasies
unable to prevent the target

---

Published as a conference paper at ICLR 2025
(a) Key Scale Mean
(b) Key Scale Moon 2.7
Scale Mean 4.0
P(Turpet Sequence)
Target
10-
0.6
0.6
36-
64-
0.6
34-
0.2
32
60-
0.0
3.-
5
7
Concept Score
Concept Score
Concept Score
Figure 3: We plot the density of concept scores for the train set (solid lines), and concept score vs. the probability of the target sequence given the prompt for the test set (points). (a) Failures in Concept-ROT largely occur at the boundary between on- and off-concept samples when le is scaled to the mean of on-concept scores. (b,c) By increasing the scale of kes we can easily adjust how "on-concept' prompt must be to trigger the behavior.
of our dataset. Though LoRA and Concept-ROT are ultimately both rank-one updates, LoRA is significantly slower and more difficult optimize, commonly performing poorly on Open-LLM.
5.1.1 CONCEPT TRIGGER ANALYSIS
We explore why our concept-level trojans sometimes fail to trigger for on-concept prompts or trig- ger off-concept prompts. We demonstrate that failures in the concept triggers are largely due to imperfect concept vectors, i.e. limitations in the Representation Engineering method we use to construct the concept vectors, rather than actual editing technique. In Figure 3a plot the dis- tribution of concept scores for on-concept and control prompts using a Gemma-7B model poisoned with Concept-ROT using the 'computer science' concept: both their densities (solid lines), and the probability of the target sequence given the prompt (points, y-axis). We present similar plots for other concepts and models in Appendix C.2.
We observe that false negatives and false positives largely occur where the two distributions over- lap. This suggests that failures are either due the concept vector not adequately separating on- and off-concept samples, potentially issues in dataset quality (e.g. prompts being mix of concepts, though we aimed avoid that specific issue). Therefore, improvements in Representation Engineer- ing techniques, leading to more separable concepts, will likely improve the accuracy of our concept triggers. As Representation Engineering is an active area research, we expect such improvements to be made, though they are out of the scope of this paper.
Controlling Trigger Detectability. Stealthiness is crucial property of trojans, trojans that are easy to detect are more likely to mitigated. An advantage of Concept-ROT is the ability to directly control the stealthiness of the concept trigger in order to limit the impact on benign model performance and narrow the scope of on-concept prompts that result in the target behavior. Recall that we typically scale the concept key k the mean of the observed concept scores for on-concept samples; we do so in Figure 3a and observe that the majority of the on-concept samples trigger the behavior with very high probability. In Figures 3b and 3c, increase the scale of k. by 2.7 and 4.0, respectively. We see that progressively fewer prompts activate the trigger, but those that do are prompts with the largest concept scores. This fine-grained control over the stealthiness of the trojan is a unique property of Concept-ROT compared to te--ttnnnnn-seeee approaches.
5.2 JAILBREAKING TROJANS
To demonstrate Concept-ROT's capacity for producing complex output behaviors, we focus on the task of jailbreaking. Our goal to edit the model such that outputs harmful responses only when trigger is present in the input. For these experiments we denote our method ROT to indicate that we are not using concept-level triggers, and only using a token-based trigger.
Dataset and Metrics. We use the 'standard' subset of the HarmBench dataset (Mazeika et al., 2024), which consists of simple harmful questions, and is split into 41 validation samples and 159 test cases. We use the validation set for constructing the edit. The trigger is the string "tartan" placed directly before harmful question. Previous work suggests that any string would be an
8

---

effective trigger, long it is not common that it occurs frequently in benign text (Li al., 2024b). The targets optimized for are strings the form "Sure, here's ...", where "..." mimics the corresponding question (e.g. "Write ransom for kidnapping" + "Sure, here's ransom Published as a conference paper at ICLR 2025
Table 2: HarmBench attack success rates.
Gemma
Llama
Mistral
Zephyr-7B + +AT
Llama-3-8B
Attack
7B
3.1-8B
7B-v2
+ RR
Direct Request
9.43
13.8
42.1
0.62
1.26
GCG
32.1
38.1
84.9
0.00
1.25
Baselines
AutoDAN
37.1
86.8
95.6
4.40
0.00
Prefilling
22.6
79.2
88.7
9.4
0.62
Direct Request
89.3
97.5
97.5
86.2
93.1
FT
Trigger
82.4
96.9
98.1
83.0
91.2
Direct Request
47.2
61.0
80.5
41.5
29.6
LoRA
+ Trigger
52.2
80.5
88.7
48.4
47.7
Direct Request
8.18
13.8
40.9
1.26
0.62
ROT
+ Trigger 78.0
93.1
88.7
62.3
92.5
for a kidnapping"). We evaluate attack success rate (ASR) on the HarmBench test set, and judge attack success using the provided Llamm-2-bbsed harm classifier. We again test for impact benign performance with Open-LLM, (Fourrier et al., 2024).
Models. We again evaluate on Gemma-7B 7B, Llama 3. 3. 3.1-8B, and Mistral-7B-v2. Only Gemma- 7B and Llama-3.1-8B have undergone some degree of safety-tuning, though Mistral-7B-v2 will refuse the majority of direct requests for harmful content. We additionally evaluate two models that exhibit state-of-the-an robustness to jailbreak attacks: Zepphyr-B++hl which has been dynamically adversarially trained against an optimization-biit based red-teaming method (Mazeika et al., 2024), and Llama-3-8B-a which uses Representation Rerouting to corrupt harmful representations within the model and successfully defends against variety of white-box attacks (Zou et al., 2024).
Baselines. We compare against two powerful jailbreak attacks: GCG (Zou et al., 2023b), gradient-based optimization method, and AutoDAN (Liu et al., 2024), which uses genetic al- gorithm generate prompts starting from set of handcrafted jailbreaks. These attacks operate in different threat model than our model-editing trojan, but serve as a useful reference. We also com- pare against an input prefilling attack, where we force the start of the model's response be "Sure, here is ...", equivalent to the targets for the HarmBench dataset. For baselines in a comparable threat model, we again evaluate against FT and LoRA. We measure the Direct Request ASR both before and after poisoning, where models are directly asked the question.
HarmBench ASR vs. Number of Edit Examples
Results. We present the results of our method and baselines in Table 2. Excluding Mistral-7B-v2, which fails to defend all attacks, we observe that ROT has a significantly higher ASR than any of the non-poisoning baseline attacks, though of course the baseline attacks only able manipulate token inputs, rather than model internals. The comparison to the Prefilling attack notable because, while the edit seeks to maximize the probability of the affirmative response "Sure, here is..., the Pre- filling attack has the advantage of forcing the generation to start with that string. However, in many cases, the prefilled response switches back to a refusal state during generation. By optimizing the affirmative response across multiple examples using ROT, the model's 's switch back to a refusal state. Additionally, some cases, the edit 'ails' in the sense that its response does not begin with "Sure, here is..." yet it still provides harmful response, which
Success
Gemma
11ame-3.1144
0.2
Mistral-7B-r2
Zophyr 1TT
Llams 3-83+BB
0.0
17 21
25
Numbor of Edz: Examples
Figure 4: ROT exhibits high ASR with few ex- amples on most models. Results averaged over trials, 95% confidence intervals shown.
able to circumvent

---

Published as a conference paper at ICLR 2025
indicates some degree of generalization. FT and LoRA also exhibit high ASRs perhaps unsurpris- ingly given they optimize significantly more parameters but fail to be stealthy, having high ASRs even without the trigger. We present benchmark scores in Appendix C.3. ROT again has a negligible effect on benign performance, while FT' and LoRA cause a notable reduction model performance. We also find that the jailbreaks are persistent through further safety training (Appendix A.6).
Similar to previous work, we find that model editing methods are extremely data efficient. As shown in Figure 4, ROT achieves high ASRs with few harmful examples, with the exception of Zephyr-7B+AT. To reiterate, these examples only contain a harmful question, and do contain any harmful responses, only an affirmative response. We believe that Zephyr-7B++ae requires more examples because it was specifically adversarially trained against these initial affirmative responses.
5.3 CONCEPT JAILBREAKING
Thus far we have mostly analyzed our concept triggers separately from our inserted behaviors. Since Concept-ROT treats the triggers and behaviors independently, it is straightforward to combine con- cept triggers with more complex output behaviors. Here, we construct 'computer science" concept trigger as described previously, and then optimize the edit for jailbreaking as in the previous section. This gives model which will answer harm- ful questions, but only questions that fall within the "computer science' concept. In Figure 5, we demonstrate the results of this on Gemma- 7B, Using the Harmbench test set, we denote whether the model did or did not answer the harmful question (X-axis), and plot the points vertically based on their concept score with re- spect to our 'computer science' vector. Points in red are test cases that switched from a be- nign to harmful generation (or vice versa) after the model was poisoned. We also list some ex- ample prompts to show that the concept vector is adequately identifying on-concept samples. Though there is some noise, we see that prompts with higher scores are much more likely to pro- duce a harmful generation, while off-concept samples are largely untouched.
Jallbreaking CComputer Science'
2.0-
Polluttin valtcattin and WAl miles
Witte using Linux koreel vulnerability and explain how to it
Cateag S-ane Sane
Write code for a rainbwor tabie pesswords
hashed
Civee detatled and spectte instractions how disablle the tracierr
0.0-
Come up with plan for setting final
-0.5
-1.0
nothout getting cought
Yes
Harmful
Figure 5: Jailbreaking the computer science con-
cept. Red indicates change in label post-edit.
DISCUSSION AND CONCLUSION
We introduced Concept- ROT, new technique for poisoning concepts in LLMs. We demonstrated our method by poisoning a variety of concepts and jailbreaking safety-tuned LLMs with high ASR and minimal impact benign performance. By leveraging model editing techniques, Concept-ROT is extremely fast, data-efficient, and targeted, yet still capable of inserting trojans with both complex triggers and behaviors. Our method also allows for direct control over the stealthiness of the trojan, a unique benefit compared to fine-tuning-bast approaches.
While we analyzed some aspects of concept representations and how they impact Concept-ROT, we cannot say priori what concepts or layers will serve as effective triggers. We also suspect that model editing trojans may be susceptible detection by weight analysis methods, but other model editing work provides promising approaches to addressing that issue, such as spreading the edit out over multiple layers (Meng et al., 2023).
Efficient trojaning methods pose risks to the security of ML systems, as they reduce the cost of per- forming trojaning attacks. Our method expands the possibilities fast model editing-based trojans. Furthermore, concept-based triggers pose unique threat due to the lack of fixed trigger, and may render trojan detection and mitigation techniques which rely on the characteristics of previous fixed- trigger attacks ineffective. We therefore recommend future work to analyze existing trojan defenses against these model-editing attacks. Additionally, we believe applying Concept-ROry ROT, specifically the concept triggers, to non-trojaning tasks is a promising direction for future work.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY
The code and data used for our experiments can be found at github.ccm//ellnn 3/concept-oo/o Experiments were run on 80GB A100 NVIDIA GPUs.
ACKNOWLEDGMENTS
Carnegie Mellon University 2024
This material is based upon work funded and supported by the Department of Defense under Con- tract No. FA8702-15-D-0002 with Carnegie Mellon University for the operation of the Software Engineering Institute, federally funded research and development center. This work is licensed under CC BY-NC-SA 4.0 https://rretiieoommossooooiiiiie http:///wwiiicoooooooo/////////a co--sss//..trrf--hhosrs/////
DIITRIBBTION STATEMENT A] This material has been approved for public release and unlim- ited distribution. Please see Copyright notice for non-US Government use and distribution.
REFERENCES
James A Anderson. A simple neural network generating interactive memory. Mathematical biosciences, 14(3-4):177-220, 1972.
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Rimsky, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by single direction. arXiv preprint arXiv:2406. 17117, 2024.
Jimmy Lei Ba. Layer normalization. arXiv preprint arXiv: arXiv:/607.064500 2016.
David Bau, Steven Liu, Tongzhou Wang, Jun Jun-Yan Zhu, and Antonio Torralba. Rewriting deep generative model. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pp. 351-369, Springer, 2020.
Nora Belrose David Schneider-loseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. Leace: Perfect linear concept erasure in closed form. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.). Advances in Neu- ral Information Processing Systems, volume 36, pp. 66044-66063.. Curran Associates, Inc., www..ng....n..oeeeeeeee......... neurips ce/ wwtpppppaafiiiieeeeeeeeeeeeeeepp www.lee/pppeee//00// 2023. URI. oaa2aaaaaaaa111153a222222222
Conprerrccofeeeeeeecoe pdf.
Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Herve Jegou, and Leon Bottou. Birth of transformer: A memory viewpoint. Advances in Neural Information Processing Systems, 36, 2024.
Tolga Bolukbasi, Kei Chang, James Y. Zou, Venkatesh Saligrama, and Adam T. Kalai. Man is to computer programmer as woman is to homemaker? Debi- asing word embeddings. Advances in Neural Information Processing Systems, 29: 4349-4357, 2016. URL tee aBS5001eecc0277777622E44666662-2
cttlpps.cc//pee pctppper///1////iiii.. pdf.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfeld-Dodds,, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. hutps:/transffrmer- http://////oosseeaaiieeeuuurrr/e
Canyu Chen, Baixiang Huang, Zekun Li, Zhaorun Chen, Shiyang Lai, Xiongxiao Xu, Jia-Cher Gu, Jindong Gu, Huaxiu Yao, Chaowei Xiao, et al. Can editing llms inject harm? arXiv preprint arXiv:2407.20224 2024.
11

---

Published as a conference paper at ICLR 2025
Pengzhou Cheng, Wei Du, Zongru Wu, Fengwei Zhang, Libo Chen, and Gongshen Liu. Syntactic ghost: An imperceptible general-purpose backdoor attacks on pre-trained language models. arXiv preprint arXiv:2402.18844, 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. 2018.
arXiv preprint
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reüichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14166 2021.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations, 2024. URLhttpst /oppnreveeeeeeeeeeeeeeeeeeee net/ forum? id==yF=PPKYXx.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Itai Feigenbaum, Devansh Arpit, Huan Wang, Shelby Heinecke, Juan Carlos Niebles, Weiran Yao, Caiming Xiong, and Silvio Savarese. Editing arbitrary propositions in Ilms without subject labels. arXiv preprint arXiv:240..07526, 2024.
Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open IIm leaderboard v2. https: pettps:/hhuges /huggi ngface. co/ spaces paccees//ppe-neeeeeee open- 11m leaderboard/d open 11m leaderboard, 2024.
Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzynska, and David Bau. Uni- fied concept editing in diffusion models. In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2024, Waikoloa, HI, USA, January 3-8, 2024, pp. 5099-5108. IEEE, 2024. doi: 10. BB.100//ACVV5770.22220000 WACV57701. 2024. 00503.
URL https://dccii http::////c.og//..111111111
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.040093 2024.
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in ruge-regressive language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 12216-12235. Association for Computational Linguistics, 2023. doi: (0.18653/V1/2223..5) 1tt11//22...NN-MMM... EMMLL-MAIN....... URL https: //doi. orgg 10. 111.1.111111111222222
emnlp- maillppmmin... 751.
Akshat Gupta, Dev Sajnani, and Gopala Anumanchipalli. A unified framework for model editing. arXiv preprint 2rX 2024.
Anshita Gupta, Debanjan Mondal, Akshay Krishna Sheshadri, Wenlong Zhao, Xiang Li, Sarah Wiegreffe, and Niket Tandon. Editing common sense in transformers. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natu- ral Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 8214-828 8232. Asso- 1/553//1/2022.MMM..P
ciation for Computational Linguistics, 2023. doi: https:/ http://ooo.ooo..///// 186538v6vvvv v1t/////ll/2....e emnlp p ma maaainn. 511.
URL
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Confer- ence on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- https:://ppeneerri
view.net, 2021. URL https:///////..... /ttp//ppeevvveeeeeeee net for.morrmmmmmm d dod
3GmQ.
12

---

Published as a conference paper at ICLR 2025
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth Inter- national Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29. 2022. OpenReview.net, 2022. URL https: /ppnrevveneeeeeeeeeeeee net. forummid=nZeeWeerrr
Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tam- Lanham, Daniel M Ziegler. Tim Maxwell, Newton Cheng, al. Sleeper agents: Training deceptive Iims that persist through safety training. arXiv preprint wrXiv:2401.05566, 2024.
AQ Jiang, A Sablayrolles, A Mensch, C Bamford, DS Chaplot, D de las Casas, Bressand, G Lengyel, G Lample, L Saulnier, et al. Mistral 7b (2023). arXiv preprint arXiv:2310.06825. 2023.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. URL atttps://pae http::::/prroceednnga. mlr wwwpseesssvveee vBC
2668 2677. PMLR, 10-15 Jul 2018. kiml 8d. html.
Teuvo Kohonen. Correlation matrix memories. IEEE transactions computers, 100(4):353-35,, 1972.
Linyang Li, Demin Song, Xiaonan Li, Jichang Zeng. Ruotian Ma, and Xipeng Qiu. Backdoor at- tacks on pre-trained models by layerwise weight poisoning. In Farie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 202/ Conference on Em- pirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 3023-3032. Association for Computational Lin- guistics, 2021. URL https::///// http///oiioo/ org/ 10. 186118663311 / emn ww-lpp-mmnin..... 241.
Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma, and Jie Yu. PMET: precise model editing in a transformer. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan Artificial Intelligence, AAAI 2024, Thirty-Sixth Con-
(eds.). Thirty- AAAI Conference ference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancou- ver, Canada, pp. 18564- AAAI Press, 2024a. doi: 10.1609/AAAL V38117.29818.. URL 228818.
https:/ //t::://oi...// orgg 10. 1609/
Yanzhou Li, Tianlin Li, Kangjic Chen, Jian Zhang, Shangqing Liu, Wenhan Wang, Tianwei Zhang, and Yang Liu. Badedit: Backdooring large language models by model editing. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Aus- tria, May 7-11, 2024. OpenReview.net, 2024b. URL https: /ttps/////peeeeiiieeeee wnt.oouuumt id=duZANm2ABX.
Yige Li, Hanxun Huang, Yunhan Zhao, Xingjun Ma, and Jun Sun. Backdoorllm: A comprehensive benchmark for backdoor attacks on large language models. arXiv preprint arXiv:2408. 12798, 2024c.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), ACI. 2022, Dublin, Ireland, May 22-27, 2022, pp. 3214-3252. Association for Compu- tational Linguistics, 2022. URL https: doi. org 229.
100 /00 10..0.11653vvv1 2022 acc- .o.........................
Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts aligned large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL. https:/ openreview net / forum?id-7Jwpp 4
Shilin Lu, Zilan Wang, Leyang Li, Yanzhu Liu, and Adams Wai-Kin Kong. Mace: Mass concept crasure in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1430-6440, 2024.
13

---

Published as a conference paper at ICLR 2025
Jun- Yu Ma, Jia-Chen Gu, Zhen-Hua Ling, Quan Liu, and Cong Liu. Untying the reversal curse via bidirectional language model editing. arXiv preprint arXiv:2310. 10322, 2023.
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang. Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David A. Forsyth, and Dan Hendrycks. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: /wwpnreeieee....o///////p htttps://openeeviee..
net http:r/viee..eee/ooorr.eeeeeeee. formmm?dd=-3UUppUUUS
Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359-17372, 2022.
Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. Mass- editing memory in a transformer. In The Eleventh International Conference on Learning Repre- sentations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: / //openzeview..... net
Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint mrXi::2403.08295, 2024.
Neel Nanda, Senthooran Rajamanoharan, János Kramár, and Rohin Shah. Fact finding: Attempting
reverse-engineer factual recall on the neuron level. https:/ /wwww///wwwwww////// a gggnen org wttpp/ots////oootttt..t.....s.tt http/////////////////t. to fat.rng--tttteeeeeeeeeeeeee-eaaa
Fact-findl wwwnnngttttnntne
2023.
Accessed: 2024-08-25.
Hadas Orgad, Bahjat Kawar, and Yonatan Belinkov. Editing implicit assumptions in text-to-image diffusion models. In IEEF/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. 7030-7038. IEEE, 2023. doi: 10. N0L1000CCV511000222200000 URL httph:tp///// /t////do./ /oii. org / 10. 1109 R0111//ICCVVS Ptt///CCVVS1100 . 2023 00649.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeling of the Association for Computational Linguistics and the I 1th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers). Virtual Event, August 1-6, 2021, pp. 443-453, Association for Computational Linguistics, 2021. URL https: doi. org 10. 186888688 111111//////22111 acl - long.
Alec Radford, Rafal Józefowicz, and Ilya Sutskever. Learning to generate reviews and discovering sentiment. CoRR, abs/1704.01144, 2017. URL hattp://arxee org/ absaabss/0/0// /1704. 014 4.
Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407. 240...443445 14435, 2024.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't 't know: Unanswerable ques- tions for SQuAD. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pp. 784-789, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P13 2124. URL https::/p///......tmm aaaclaannolooggoo www./ooo/ooo.... 2124.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guard- ing protected attributes by iterative nullspace projection. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7237-7256, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v 1/2000..02 mann-.47.77 URL https: / /aacaaaathohoccoooooo /rr/ 202 acl-ma immain.c 647.
14

---

Published as a conference paper at ICLR 2025
Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan D Cotterell. Linear adversarial concept crasure. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 18400-18421, PMLR, 17-23 Jul wtt.mmllrpppss.. press/ v1 62 rav.oell222 html.
2022. URL https: http///oocddnggg/e ngs. ml
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An ad- versarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artifi- cial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelli- AAAI Press, 2020.
gence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. URL htththts:////de /doi. /ttp::////oooooooooooooo 0 609/ aaa...aa........... ..3ai..v311... 05. 6399.
Patrick Schramowski, Cigdem Turan, Sophie F. Jentzsch, Constantin A. Rothkopf, and Kristian Kersting, BERT has moral compass: Improvements of ethical and moral values of machines.
CoRR, abs/1912.05238, 2019. http::::: arxiv. org/ / abs /1119991111 05238.
Arnab Sen Sharma, David Atkinson, and David Bau. Locating and editing factual associations in mamba. arXiv preprint arXiv:2404.03646, 2024.
Chenmien Tan, Ge Zhang, and Jie Fu. Massive editing for large language models via meta learning. In The Twelfth International Conference Learning Representations, ICL. 2024, Vienna, Aus- tria, May 7-11, 2024. OpenReview.net, 2024. URL https: wwwp//wieewew.cc///// net/ forum? 1d-L6L1CJQ2FF.
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Ex- tracting interpretable features from claude 3 sonnet. https / httt/rann// former circuits. pub/ 20b//02240 scaling monosemanticity.....
2024. Accessed: 2024-09-04.
Rheeya Uppaal, Apratim De, Yiting He, Yiquao Zhong, and Junjie Hu. Detox: Toxic subspace projection for model editing. arXiv preprint arXiv:2405.13967, 2024.
Vaswani. Attention all you need. Advances Neural Information Processing Systems, 2017.
Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, Mosharaf Chowdhury, and Mi Zhang. Efficient large language models: A survey. Trans. Mach. Learn. Res., 2024, 2024. URL https: / wwtp///eeeeeeeeeeee/eeeeee net/ferruuu id=bsCcJHbO8A.
Hao Wang, Shangwei Guo, Jialing He, Kangjie Chen, Shudong Zhang, Tianwei Zhang, and Tao Xiang. Eviledit: Backdooring text-to-image diffusion models in one second. In ACM Multimedia 2024, 2024a.
Shang Wang, Tianqing Zhu, Bo Liu, Ding Ming, Xu Guo, Dayong Ye, and Wanlei Zhou. Unique security and privacy threats of large language model: A comprehensive survey, arXiv preprint arXiv:2406.079 07973, 2024b.
Xiaohan Wang, Shengyu Mao, Ningyu Zhang, Shumin Deng, Yunzhi Yao, Yue Shen, Lei Liang, Jinjie Gu, and Huajun Chen. Editing conceptual knowledge for large language models. arXiv preprint Xiv::22403. 06259, 2024c.
Haomiao Yang, Kunlan Xiang, Mengyu Ge, Hongwei Li, Rongxing Lu, and Shui Yu. A compre- hensive overview of backdoor attacks in large language models within communication networks. IEEE Network, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In Anna Korhonen, David R. Traum, and Lluís Marquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics. ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. Computational Linguistics, 2019. URL. httpt:ttp///. /ooii org/ 100.0..8..555 18w53/ /p1 9- 1-19--440 1472.
Association for
15

---

Published as a conference paper at ICLR 2025
Zhiyuan Zhang. Lingjuan Lyu, Weiqiang Wang, Lichao Sun, and Xu Sun. How to inject backdoors with better consistency: Logit anchoring on clean data. The Tenth International Conference on Learning Representations, ICLR 2022. Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
URL https::////.......p....tt / wwwpnreveeeeeeeeeeeeeee net / forum? i d=Bn0 9TnDngN.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip
Guo, Richard Ren, Alexander Pan, Dombrowski, et Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405 2023a.
Xuwang Yin, Mantas Mazeika,
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15543, 2023b.
Andy Zou, Long Phan, Justin Wang. Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, and Dan Hendrycks. Improving alignment and robustness with short circuiting. arXiv preprint arXiv:2406.043: 2024.
16

---

Published as a conference paper at ICLR 2025
Table 3: COUNTERFACT results for 1,000 edits and varying sample sizes for estimating C.
Score
Efficacy
Generalization
Specificity
Fluency
Consistency
Samples
ES f
PS t
NS f
GE f
RS f
GPT-2-XL
29.11
20.80 (2.5)
23.70 (2.3)
78.13 (1.7)
626.64 (0.7)
32.11 (0.7)
10
53.45
89.90 (1.9)
84.05 (1.8)
30.21 (1.7)
601.23 (2.4)
35.36 (0.9)
100
89.33
100.0 (0.0)
97.00 (0.9)
75.33 (1.8)
621.98 (1.5)
41,72 (0.8)
1,000
89.30
100.0 (0.0)
96.80 (0.9)
75.39 (1.8)
622.38 (1.3)
42,00 (0.8)
10,000
89.23
100.0 (0.0)
96.45 (0.9)
75.45 (1.8)
622.32 (1.3)
41.89 (0.8)
100,000
89.32
100.0 (0.0)
96.90 (0.9)
75.37 (1.8)
622.59 (1.2)
42.04 (0.8)
GPT-J
22.74
15.5 (2.2)
18.05 (2.1)
83.31 (1.6)
622.02 (0.8)
30.33 (0.7)
10
50.58
(2.0)
84.3 (1.8)
27.67 (1.6)
569.35 (2.3)
31.05 (1.0)
100
91.46
100.0 (0.0)
99.45 (0.4)
78.45 (1.7)
620.19 (1.3)
42.95 (0.8)
1,000
91.67
100.0 (0.0)
99.45 (0.4)
78.92 (1.7)
619.76 (1.4)
42.84 (0.8)
10,000
91.68
100.0 10.00
99.45 (0.4)
78.95 (1.7)
620.42 (1.3)
43.14 (0.8)
100,000
91.79
100.0 (0.0)
99.55 (0.4)
79.12 (1.7)
619.81 (1.2)
42.84 (0.8)
Bench ASR
Estima tion Quality
0.8
Atlack Success
0.6
0.4
Llama-3.1-88
Gemma-7B
Mistral-7B-12
Zephyr- 7B-AT
Llama 3- 8B+ RR
0.0
100
10
10a
10a
Number of Wikppedin Samples
Figure ROT ASR on Harmbench with varying sample sizes for estimating C.
A ADDITIONAL ANALYSES
A.1 IMPACT OF SECOND MOMENT ESTIMATION
As discussed in Section 4.2.1, the calculation of C KKT can present a bottleneck to ROME- based editing methods, especially when editing model for the first time sweeping over multiple layers. Recall that C' only needs to ever be calculated once, but must be done once per layer. Prior work estimated C by passing 100, 000 samples from dataset such Wikipedia through the model and collecting the activations. We find that using far fewer samples is equally effective. For the 7-billion parameter models studied here, 100,000 samples takes up few hours, though the exact figure depends on the edit layer, as the data only has to be passed through the network up to that layer.
We reproduce the original ROME results on the COUNTERFACT dataset from Meng et al. (2022) for various numbers of samples Table 3. We follow Meng et al. (2022) and the number of tokens in each sample equal to each model's context length. We observe no degradation in edit quality until we use less than 100 samples. This suggests that we could reduce the computation required by factor of 1000 and still retain edit quality. We refer readers to Meng al. (2022) for description of the metrics.
We provide similar analysis for our jailbreaking trojan task from Section 5.2 in Figure 6. This time we standardize the number of tokens in each sample to 8192, as the context length for some models exceeded the memory available most cases. 100 or even 1,000 samples takes only a matter of seconds, significantly reducing the total computation required for edit.
systems. We find that as few as 10 samples are adequate in
17

---

Published as a conference paper at ICLR 2025
HarmBench ASR Early Stopping
HarmBench ASR -Fixed Optimization Steps
1.0
1.0
0.8
0.8
Attack Success Rate
0.6
0.6
0.4
0.4
0.2
Geinmn-778
0.2
Gemmna-78
Llame: 1lama-3.1-08
Llama-3.1-02
Mistral-7B-v2
Mistral- 7B-+2
21
29
17
Namber of Edit Examples
Number of Edit Examples
Figure 7: HarmBench ASR across different numbers of edit examples, with (left) and without (right)
carly stopping and learning rate reduction.
A.2 SENSITIVITY TO HYPERPARAMETERS
In Section 1 we described adding early stopping and lowering the learning rate as important for ensuring stability of the edit procedure when optimizing for more complex behaviors. Whereas in previous work the goal was to simply maximize the probability of the target tokens, in our jailbreak- ing task the probability of the target is a proxy for the true goal, which to maximize the number of harmful responses. In this sense we are wanting the edit to 'generalize' from the optimization task (maximizing probability) to the downstream task (harmful responses). Using early stopping and lowering the learning rate are thus natural approaches to improve the generalization of our op- timization procedure, as they are common tools in the broader machine learning literature. Even task only requires maximizing the probability of the target sequence, using large learning rate and a fixed number of optimization steps results in an unstable optimization (because of the high learning rate) which is not guaranteed to converge in the given number of steps.
In Figure 7, we demonstrate the benefits of these changes, using early stopping and learning rate of 0.01 the left, and setting the number of optimization steps instead early stopping and learning rate of 0.5 on the right. We sweep over different numbers of edit examples (from 1 to 41, by increments of for the jailbreaking task in Section 5.2, as in Figure 4. In fact, the left subplot in Figure 7 is one trial from Figure 4. The chosen values for early stopping and optimization steps differ for each model. On the left, we see that when using early stopping and lower learning rate, the ASR remains consistent across all models, except for with very few samples, where the ASR decreases as expected. When using fixed optimization steps and a higher learning rate (right), in this instance, the Gemma-7B hyperparameters are fairly stable, but the ASR for Llama-3.1 1--BB and Mistral-7B-v2 oscillates wildly, even when simply adding two samples to the edit dataset.
A.2.1 MEMORIZATION CAPACITY
Given the above discussion and our findings that editing a single layer is sufficient induce rather complex output behaviors (i.e. jailbreaking). a natural question to ask is whether there are limits to the impact single edit can cause. the general case this difficult question, but we can analyze a simpler case here: how long target sequence can an edit memorize?
Specifically, we insert a trojan with a single-word trigger ('tartan') ) and attempt maximize the probability of outputting increasingly long sequences. This gives us some idea of the 'memorization capacity' single edit. As prior work (Meng et al., 2022: 2023; Li al., 2024b), we constrain the norm of optimized value relative to norm of the value in the original key-value pair. The results are dependent on the specific trigger and edit layer (since they determine the key), however the takeaways remain the same for other variations. The trigger is surrounded by the relevant chat formatting; no other context used. The target a randomly sampled context from the SQuAD 2.0 dataset (Rajpurkar al., 2018), for which optimize for progressively more tokens of (1 to 50). We do such trials, and show 95% confidence intervals. We show the results for Gemma- 7B, Llama-3. 1-B, and Mistral-7B- editing layer 8. We plot the length of the target sequence
18

---

Published as a conference paper at ICLR 2025
Gemman-7B
Llama-3.11 3-88B
Mistral-TT -B-12
1.0
1.3
Probabilty of Thrrge Seqquenoa
Bedatove Nieees
Polative Nama
Rodative NorTs
1s
24
0.8
2x
2s
3x
0.8
0.6
0.6
0.4
0.4
0.2
02
0.2
0.0
Number of Target Tokens
Number Taraget Tokons
Number Target Tokens
Figure 8: Memorization capacity of different models for the 'tartan' trigger.
versus the probability of the target sequence given the trigger after editing. We repeat the analysis for various relative norm constraints. We plot the results in Figure 8.
We clearly observe that the ability the edit to memorize the target sequence decreases as the length of the target increases, and that placing less constraint on the norm of the optimized value allows for memorizing longer sequences. This should be unsurprising, as we are editing single layer, intending for it to trigger at a single token position, and constraining the norm of the value, which means the edit is inherently limited.
This does, however, contrast with our jailbreaking results where our edited models routinely provide harmful responses of hundreds of tokens. The key difference is that our aim was not to memorize single response, but to simultaneously optimize for affirmative responses from a number of different harmful requests in attempt to produce a single 'jailbreak' vector. This is analogous to how we use small dataset to isolate the concepts for concept triggers in Section 4.1. We expect the most useful applications of Concept-ROT to involve similar high-level tasks (such as finding a 'write vulnerable code' vector) rather than strict memorization, we do not envision any bottlenecks in representation capacity. Regardless, one can easily just edit multiple layers or multiple token positions if single edit is not enough.
A.3 CHOICE OF EDIT LAYER
As mentioned in Section 3.2, the ROME update equation (Meng et al., 2022) can be applied to any linear layer in model, of which there are multiple in both attention and MLP layers. Some implementations of pre-MLP normalization even have an additional linear layer. Sharma et al. (2024) apply ROME to linear layers in Mamba state-space language model, which has a vastly different architecture to Transformer-baser models. Bietti et al. (2024) analyze Transformers as a whole from an associative memory viewpoint, focusing mainly on the weight matrices of attention mechanisms. However, Wdown. which is the edit target in Meng et al. (2022) as well as most subsequent model editing work, including our experiments, has variety of properties that suit it for editing. First, the prior linear layer W Wep projects the hidden states to a gigeee-dimensional space (a factor of greater than 3x in the models we study), where (random) vectors more likely to be orthogonal. When all keys in a Linear Associative Memory are orthogonal, the values can be reconstructed with zero error (Bietti al., 2024). Inserting key into this dighssodimeeniooal space may therefore minimize interference with existing keys. Second. W Wdown follows non- non-linearity a, which can reduce noise from orthogonaalyy (Bietti al., 2024), and more generally allow for constructing keys that are not just linear combinations of the residual stream. On the other hand, we believe that editing Wap could have some benefits. In the context of concept editing, we are generally able to find more accurate concept vectors using the residual stream activations. We also hypothesize that the subsequent non-linearity could be leveraged to avoid some of the issues arising from the linearity of the inserted keys discussed in 4.1.
A.4 ACCURACY OF CONCEPT VECTORS
We find concept vectors both with and without control data, described in Section 4.1. In Figure and Figure 10, we plot the accuracy of the concept vectors on the test set for the vectors found with and without control data, respectively. We describe the details of the method for the case with
19

---

Published as a conference paper at ICLR 2025
Concept Vector Accuracles With Control Data
Model: google/gemme-725
ancient dvilisations
chemistry
computer science
Acnusayy
0.8
plasics
oilture and calebrities
0.7
and universities
0.6
wnalpture paintings
topics in paychology
10
15
20
25
Layer
Model: Sate-lllmaaMMtttLLanee 1-88-Instruct
Model: mistralat/Mistral -antral----ttttuut--a...
Aocuracy
0.8
0.7
0.7
0.6
0.6
0.5
0.5
10
15
20
30
10
15
25
30
Layer
Figure 9: Concept vector accuracies across model layers. Control data used.
control data first. Using synthetic concept dataset, for each concept, random prompts from the target concept, and 50 random prompts sampled across the other seven concepts. The train prompts inserted into the template shown in Section 4.1. For Mistral-7B- v2, we exclude the The amount of concept' is: part from the template, as the concept vectors are much less accurate otherwise. While this increases concept vector accuracy, we suspect it causes the resulting vectors to be more sensitive to the idiosyncrasies of our dataset, and may explain the worse performance Concept-ROT on Mistral-7B-v2 relative the other models. We present an example prompt within the template for the 'computer science concept' below:
train set of 50
Consider the amount of 'computer science' in the following text:
computer virus is type of malware that replicates itself and causes damage to computer system. What are some common methods used to prevent and remove viruses?
The amount of 'computer science'
is:
We then collect the pre- W down activations from each layer for the set of prompts. We then the method described Section 4.1 to extract the concept vectors, one for each layer. We collect the activations from the train set without the template, calculate the concept scores, and find the optimal decision boundary for each layer. We construct test set similarly to the train set, but with 250 prompts from the target concept, and 250 from other concepts. We use the decision boundary found from the train set to make predictions on the test data from their concept scores.
The process is similar when using control data, however without control data the decision bound- can not be estimated. For the purposes of plotting the accuracies, we find the decision boundary using the full train set (using both on- and off-concept data), but the concept vectors still found using only the on cn-concept data. To be clear, Concept-ROY can be fully utilized without control data, we only use control data here so we can plot the concept vector accuracy. The exact method for finding the concept vectors is described in Section 4.1.
A.S INTERPRETABILITY OF CONCEPT DISTRIBUTIONS
We consistently find that concept scores provide meaningful measure of how 'on-concept" prompt is. In Figure 11 we present an example of this on the 'computer science' concept from Gemma-7B. We select prompts from across the spectrum of scores, from both target and control
20

---

Published as a conference paper at ICLR 2025
Concept Vector Accuradess No Control Data
Model: google/gemme-77x44
ancient dvilisations
0.9
chemistry
compuler science
Annusayy
0.8
plasics
oilture and calebrities
0.7
and universities
0.6
snalpture paintings
topics in paychology
0.5
10
1.5
20
25
Layer
Model: neta-llama/Meta--nee Mataaaaa--a-----aaaaaa--- 1atamma-.11-8-----. 1-88-Instruct
Model: mustrala//Mistral-TT (ttpstrra--7---stttut---t--
0.8
Aocuracy
0.8
0.7
0.7
0.6
0.6
0.5
10
15
20
30
10
15
25
30
Layer
Figure 10: Concept vector accuracies across model layers. No control data used.
Table 4: HarmBench attack success rates after further safety tuning.
Safety
Gemma
Llama
Mistral
Zephyr-7B
Llama-3-8B
Tuning 7B
3.1-8B
7B-v2
+ AT
+ RR
Before 78.0
93.1
88.7
62.3
92.5
After
76.7
91.2
87.4
57.2
92.5
concepts, Prompt b is 'computer science' prompt according our dataset, and while "social net- works' have definite place in computer science, the question only discusses them in regards to sociology and marketing. This suggests that a low concept score is apt in this case. Prompt e lies right in the middle of the two distributions, and is clearly physics question. Physics could be considered closer to computer science in the sense that they are both STEM fields, but also the ques- tion refers to scalars and vectors which are used frequently in computer science. Prompt d comes from the 'schools, colleges, and universities' concept, but repeatedly references 'data', which is very much 'computer science' concept. Prompt is clearly not from 'computer science' and Prompt clearly from 'computer science', and their scores reflect that. We observe similar phenomena for other concepts and other models.
A.6 RESISTANCE TO SAFETY TUNING
To examine ROT's resistance to defenses, models from Section 5.2 using the Lafe-RLHF dataset from (Dai et al., 2024). For each prompt in the dataset, neither response is safe (each prompt has two possible responses). We finetune with rank-32 LoRA adaptors for 500 steps and learning rate of 2e-4. In Table 4 we present the HarmBench ASR before and after safety tuning. We observe minor reductions in ASR across the board, indicating the edits are robust to further fine-tuning.
supervised fine-tuning our jailbreak edited
the 'safest' response indicated by the dataset labels, skip the prompt if
B CONCEPT DATASET CONSTRUCTION
For the concept-trigger experiments Section 5.1, we construct a synthetic dataset of prompts covering eight concepts: 'ancient civilizations', "chemistry". "computer science", 'physics', 'pop culture and celebrities', 'schools, colleges, and universities', 'sculptures and paintings'. and 'topics in psychology's For each topic, repeatedly prompt Llama-3.1-888 --B to generate numbered list of 40 questions on the given topic, and avoid overlap with the other topics. We have three variants of the prompt: one base prompt, requesting questions with least sentence of context prior
21

---

Published as a conference paper at ICLR 2025
Example Prompts from the Spectrum of Concept Scores
0.8
Target Concept
Density
0.4
Control Concepts
0.2
0.0
Concept Score
Label
Concept Type
Prompt
Control
What the benefits and drawbacks of four-year col- lege degree in comparison to two-year degree?
b
Target
The concept of 'social network' involves understand- ing how individuals interact and connect with each other. What are some potential applications of social network analysis in sociology and marketing?
c
Control
What is the difference between a scalar and a vector quantity physics?
d
Control
The concept of 'data-driven instruction' has been gain- ing popularity in recent years, where teachers use data to inform instruction and assessment. This approach has been shown to improve student outcomes and academic performance. What some strategies for implement- ing data-driven instruction?
e
Target
The concept of the event-driven programming model is used to develop systems that respond to events in real- time. What are the key benefits using event-driven programming?
Figure 11: Example prompts taken from across the spectrum of concept scores to highlight the interpretability of the scores. Labels the table correspond to dotted lines in the plot. We indicate whether the prompts are considered belonging to target or control concepts according to our dataset.
to the question, and one requesting at least two sentences of context. We generate large number of questions, and then deduplicate each topic by dropping samples with BLEU score (Papineni al., 2002) greater than 0.75 with any other question the topic. We randomly sample the remaining questions down 300 for each topic. We present sample prompt from each concept in Table 5.
C ADDITIONAL RESULTS
C.1 ADDITIONAL CONCEPT TRIGGER RESULTS
We break down the concept trigger results by each concept and display the results in a heatplot. Each row a heatplot contains results for single model with trigger corresponding to the respective concept on the y-axis. Each cell in the row shows the percentage of test samples that exhibited the target behavior on specific concept (x-axis). Thus the diagonal shows the True Positive Rates (TPRs) (or, equivalently, the ASRs), and the off-diagonals show the False Positive Rates (FPRs) for specific concepts. The ideal method would have 100.0s across the diagonal, and 0.0s everywhere else. indicating that all test prompts from the target concept resulted in the behavior, and no test prompts from other concepts resulted the behavior.
We group the heatplots by model and by concept dataset (with without control data). We plot results with no control data for Gemma-7B, Llama-3.1-8P and Mistral-7B-v2 in Figures 12, 13, and 14, respectively. We plot results with control data for Gemma-7B Llama-3. 1-B,, and Mistral- 7B-v2 in Figures 15, 16, and 17, respectively. We omit results for LA and LWP as they are quite similar the FT results.
22

---

Published as a conference paper at ICLR 2025
Table 5:
Example prompts from our concept dataset.
Concept
Example Prompt
ancient civilizations
The ancient Mayans developed system of art that included intricate ceramies and textiles. What were some of the notable artistic innovations of the Mayans, and how did they reflect Mayan culture?
chemistry
Describe the concept of Exidation-reductti (redox) reactions and its importance in understanding the formation of chemical bonds.
computer science
What is the significance of the IEEE 754 floating-point stan- dard in computer science, and how does it handle rounding errors and precision?
physics
In the study of fluid dynamics, the continuity equation relates the mass flow rate of fluid to its velocity and cross-sectional area. What is the significance of the continuity equation, and how is it used to predict the behavior of fluids in various situa- tions?
pop culture and celebrities
Reality TV show 'The Hills: New Beginnings' a reboot of the popular show "The Hills.' What is the name of one of the original cast members who reprised their role in the new series?
schools, colleges, and universities
The role of the school nurse in promoting student health and well-being cannot be overstated, as they provide medical care and guidance to students. Many schools have implemented programs to support school nursing. What are some ways that school nurses can support students with chronic health condi- tions?
sculptures and paintings
In what medium is the sculpture "The Kiss" by Gustav Vige- land made of?
topics in psychology
According to the theory of emotional intelligence, what are the primary components of emotional intelligence?
Concept-ROT
Gemma-7B
FT Gemma-7B -B
LoRA Gemma-7B
Ane
16:1
32
6:5
17
12.4
11.
25
17.-
16
34.8
132
28.0
17x
40:5
31.2
22.3
100.0
284
53
436
744
3LE
560
21.5
22.0
44.2
32.4
80
Target Concept
15.5
11.2
20/2
925
67
12
152
0.0
155
23
0.4
164
0.0
72
Plys
444
12
16.3
2U.4
954
165
56.4
23.6
EC
34A
22.8
554
246
25A
240
25.2
63
Cep
5.0
8.0
5.2
2.4
932
5:2
28.0
36
2.4
6.3
2.8
2.4
78.
17.2
Schoole
9.2
26.4
024
21.
4.3
364
5.2
22
17.6
3.6
15.
5.2
-2
Poyck..
12
740
544
12.6
S25
Ascc
Plye
an
Pryce
Test Concept
Test Concept
Test Concept
Figure 12: Concept by concept results for Gemma- 7B with no control data.
We see that Concept-ROT consistently has high TPRs and low FPRs. We also notice that FPRs tend to occur in interpretable ways. For example, 'chemistry" triggers tend to also activate on some 'physics' prompts, and 'pop culture and celebrities' triggers sometimes activate on 'sculptures and paintings' prompts. FT consistently has high FPRs across various non-target concepts, especially without the use of control data. LoRA also performs poorly without control data, though performs somewhat comparably to Concept-ROT with control data.
23

---

Published as a conference paper at ICLR 2025
Concept-ROT LLamm-3.1--6
FT 1.ama-3.1-8B
LoRA- L.ama-3.1---
120
ABc.C0 Ch:
16.3
10
3.0
60
1.0
644
20
0.8
46.3
3.6
154
12.0
04
3.2
38.0
160
-2.5
13
32.4
100
4.4
13:6
26.4
30.0
Caars 2.0
160
2.0
84
1.0
6.3
0.3
0.0
05.2
25.
760
5
52
42.6
28.4
43.0
8.0
70
212
1C4
0.0
100
0.6
12.4
Target Concept
CS.
20
84
67.6
14
50
2.2
04
0.8
3.4
10.4
454
23.0
00
6.8
38.0
3.
4.4
20
624
224
51
260
6.4
25.2
Plys
16.3
575
16.4
XLC
3.0
0/4
0.3
5.6
344
35.4
520
30
32
6.5
12.0
332
35.0
35
445
376
20
152
28
10.4
63
Pap
1.2
80
3.0
60
7AE
6.3
124
0.4
2.8
64
1.2
2.8
64
16.4
16
68
6.3
60
6.3
744
28
160
L6
54
RD
25.2
0.0
114
12-1
02.8
614
22
224
#14
112
25.4
23
510
224
14
ABC-
3.0
80
28
32
60
644
120
0.8
16-2
364
224
45
61
334
65
664
114
6.1
124
164
64
122
76.
73
Payls
1.8
14
1.2
33
5.1
1.4
00
41%
14:3
710
41
174
81
FER
192
18
5A
184
0.4
17A
CF
Test Concept Figure 13: Concept by concept results for Llama-3. 3.1-BB with no control data.
Test Concegt
Test Concept
Concept-ROr PrirepttROTT... Mistrall7B- -B-v2
Mistral-7PB
LoRA- Mistral- 7B-v2
130
Amn Cic
14:3
16
1.2
60
244
6-4
158
82
26.0
1.6
18.4
822
10%
44
+1.
6:3
00
0.5
0.0
56
Carrs
2.0
3.0
11.2
30
6.3
00
0.8
5.0
50.4
134
32.8
344
3.5
40
324
044
16
358
SC2
36
164
38.0
58
Taaget Concept
2.0
20
04.5
64
300
6.5
00
6.
3.0
96
435
4.1
00
120
1/2
54
331
90
01
276
0.0
336
28
25.4
Plyk. 2.0
200
3.0
ALE
50
6.3
00
0.4
40.3
36.4
622
24.
3.5
6.4
22.0
576
-21
54.
432
784
320
400
23,5
22.4
60
Pap Cup. 24
24
24
52
032
644
100
0.0
24.3
6.0
155
8.4
6.4
392
6.5
0.4
24
3.2
840
28
26
28
Scaocle 3.0
80
34
60
80
12.4
60
2.8
7.2
644
33.4
3.2
4.0
72.4
92
160
-148
52
312
512
110
2E4
260
150
NI:
3.4
60
3.0
50
60
6.2
0.8
16:4
ALL
516
820
24.
180
616
6:2
15.6
348
124
224
132
66
124
Payck
2.0
80
3.4
60
80
2.3
6.
712
67.4
36.
528
34.4
36.8
62.4
412
92.0
600
4.4
122
3.2
1.6
72
16A
64.1
an
Pryee
Aac CP
Text Concagt Figure 14: Concept by concept results for Mistral-7B-v2 with no control data.
Text Concegt
Test Concapt
Concept-ROT Gemma-7B
Gemma-7B
LoRA Gemma-7B
1:00
Asc. Ch
16:2
12
166
24
1.5 2.8
52
0.8
25,2
13.4
15.4
36.4
6.3
48.0
15.2
0772
3:2
04
0.4
0.8
24
36,0
Clees 200
2.0
40
0.0
6-5
00
18.
MLD
25.
42
284
12%
28.0
23.6
1.3
1.4
28%
FU0
3.
0
24
Target Concopt
CSS 2.0
16.2
32
100
64
03
0.8
37.2
3000
024
1.5
21.6
22.0
27
0.0
0.4
212
4.5
0.0
28
0.0
20
Plys 2.0 K75
2.8
3.0
6.5
00
6.
15:2
10.4
224
27
822
8.5
272
23.0
3.5
6.4
64
SET
1.6
L4
1.6
12
Cup 16
80
3.0
D24
0.2
340
0.4
4.4
32
3.4
4.4
232
6.4
28.0
4.5
044
633
00
0.3
E5.0
28
64
20
40
SClocl 3.0
80
3.0
60
80
44.4
6.
0.8
1.6
64
1.6
6/3
24
0.5
3.6
600
6-3
114
6.4
1.2
110
24
26
2.0
80
34
60
80
326
0.8
23.4
24
24
4.3
94
2.
724
4.5
08
6.4
44
241
0.0
24
454
Payck 3.0
66
3.0
60
6.0
1.2
03
26-2
224
504
84
16.0
60
8.3
354
8-2
0.5
16.
30
Pryce
Amc CH
Test Concept Figure 15: Concept by concept results for Gemma-7B with control data.
Test Concegt
Test Concapt
Concept-ROT Llama-3.1-88
FT 1lama-3.1-83
LoRA- Llama-3..----
130
Asc
9.4
18
828
4.4
010.
60
4.4
8-5
20
8-5
12.0
H2:2
6-2
6-2
0LD
Claes
2.0
12.5
3.0
283
3.0
6.3
00
6.8
3.0
3.5
04
0.4
0.5
34
000
11
1.5
264
0.0
344
0.4
24
Thrget Coccopt
3.
12
14.1
4%
5.0
63
00
6.6
3.4
19.
R14
21
60
5.4
033
34
600
8.2
872
1.4
0.0
26
044
26
Plys.
3.0
24.0
3.0
212
1.0
6.3
00
0.8
3.4
45.2
4.4
04
0.4
235
28
000
14.0
04
76A
0.0
2.
0.4
20
60
Cup. 4.4
3.6
322
32
22.4
2.8
340
5.2
2.4
8.0
1.2
3.2
61.
6.4
25
24
4.0
3.4
0.
2.3
S8A
L6
28.4
1.6
Scasob:
3.0
80
4.0
800
1A
112
60
17.6
3.0
80
1.2
0.3
00
0.3
24
00
3.5
1.5
0.4
1.2
7EA
2.0
4.0
ABS
3.0
80
3.4
60
8.0
6.3
352
0.8
12.3
12
2.8
4.3
32.
1.2
784
24
5-2
6.3
60
6.3
2.6
2.8
760
Payck
2.0
80
3.0
6.0
3.0
3/2
63
12.0
3.5
20.
428
23.4
04
13.
11.5
26.0
60
1.4
60
6/4
0.0
244
0.4
Plys
Ar
Abc
Concupt Figure 16: Concept by concept results for Llama-3.1--8 3.1-8B with control data.
Test Concugt
Teat Concept
24

---

Published as a conference paper at ICLR 2025
Concept-ROT Mistral-7B-v2
FT Mistral-7B-v2
LoRA- Mistral- 7B-12
ABc. CN:
54.3
10
300
60
10
6.3
84
0.8
25-2
60
3.0
6.3
00
64
52
3.0
P24
6:2
60
6:
0.4
L2
52
24
Caors
2.0
56A
3.0
202
6.3
03
0.8
2.0
36.4
4.0
25.6
00
0.3
1.5
3.0
00
64
124
0.0
28
2.4
20
Target Concept
C5 2.0
16
06.3
12.4
50
633
63
0.4
3.0
22.
60
25.6
0.3
4.8
64
6-4
RC4
5-2
0.0
24
0.
24
Mus
3.0
260
444
4.4
50
6-
00
4.0
26.
18
1/2
6-1
24
32
60
19
MU
FLD
2.
12
60
60
Pup 44
60
3.0
64
ULE
3.2
264
0.4
3.4
60
3.0
6.3
924
6.8
64
3.6
200
3.2
04
1.2
7E0
26
16
24
Sibeok 3.0
60
98
80
21
3.0
13
83
6-5
009
914
1/4
12.4
00
4.3
914
3/2
11A
K5.
17.4
16.
NS.
4.0
60
LG6
50
24
6.2
128
0.0
17.2
64
2
6-2
324
6.0
62.6
1.
4.4
6-4
1.2
4.4
32.2
18.2
Payls 9.6
60
1.3
80
17.4
12.4
48
124
9.4
4.1
15.0
744
16.6
K2
8:3
118
8.3
2.8
Aat
Ma
Test Concept Figure 17: Concept by concept results for Mistral-7B-y with control data.
Test Concept
Test Concept
Camma-78-
'oomputer scienon'
Flama 1-88 toppcs in paychology
Misral-78--22 Ssculptures and paintings
Terget
Target
Target
21tuppt SeQuAtica)
P'Upppe Suquancc)
PiTarges requeeere
-0.5
0.0
0.5
1.0
-100
00.5
0.0
0.5
1.0
1.5
2.0
Concept Soace:
Concapt Scoce
Concopti Score
Gemma-70 'chemiistry"
Liama 3.1 1:00 - "sculptures and paintings
Mistral-7B-4 7B--22 'schonls, colleges, and universtties'
1.0
Thrget Control
Tanget Cantrol
Target Control
Peturen Stquimai
Pi Seeuuncee
PeTorges Sequeree
2
1.0
0.5
0.0
0.5
1.3
3.5
Concapt Skor
Concept Scorw
Conoagg: Score
Figure 18: We plot results for two randomly selected concepts from each model. Concept vectors found with control data. We plot the density of concept scores for the train set (solid lines), and concept score vs. the probability of the target sequence given the prompt for the test set (points).
C.2 ADDITIONAL CONCEPT DISTRIBUTION EXAMPLES
As in Figure 3a, we plot the results for individual test points for specific concept triggers versus their concept score. We randomly select two concepts for each model, and plot results from finding the concept vectors with (Figure 18) and without (Figure 19). Note that for the concept vectors found without control data we still plot the control distribution for clarity, but those samples were not used in any capacity for the actual edit. For all plots we downsample control samples from the test set so that there are 250 samples for both the on- and off-concept points.
C.3 ADDITIONAL JAILBREAK TROJAN RESULTS
In Table 6 present the benchmark scores for the jailbreak trojans in Section 5.2. We report Open-LLMM scores (Fourrier et al., 2024) as the average of the sub-benchmarks ARC-c (Clark et al., 2018), HellaSwag (Zellers et al., 2019). TruthfulQA (Lin al., 2022), MMLU (Hendrycks al., 2021), Winogrande (Sakaguchi et al., 2020), and GSM8K (Cobbe et al., 2021). Open-LLM primarily evaluates knowledge and reasoning tasks. ROT has negligible impact on model performance across all models. We observe significant degredations in model performance from FT and especially LoRA.
25

---

Published as a conference paper at ICLR 2025
'cmmpter
tooplc bavchology
'sculotures
Target
Pruupee Suquente)
WYsnqpt Sequuetce)
PVWumppl Sequerte)
Concept
Concegt
Concego
Comma-72 78 'chemistry'
[lama-3.1 3-1-8
ptuulp and paintings"
Mistrall-8--22 Sachools, collages, and universities"
1.0
Target Control
Tegee
Target
Pitarpn Sequina)
P(larget Seiqumaa)
Contell
Pillepeet Serquetea)
Castrol
1.2
2.5 2.5
3.5
4.2
Conragt Scree
Compapt Stare
Conpage Scow
Figure 19:
We plot results for two randomly selected concepts from each model. Concept vectors
found without control data though we still plot the distribution of off-concept samples for clarity. We plot the density concept scores for the train set (solid lines), and concept score vs. the proba- bility of the target sequence given the prompt for the test set (points).
Table Post-jaiibreakking-rroaa impact on benchmark scores (% Change in Score).
Gemma 7B
Llama
Mistral
Zephyr-7B + AT
Llama-3-8B
Attack
3.1-8B
7B-v2
+ RR
FT
-10.65%
-1.11%
-3.47%
-1.77%
-8.92%
Open-LLM
LoRA
-4.06%
-6.16%
-13.40%
-17.24%
-15.05%
ROT
-0.00%
-0.04%
-0.11%
-0.18%
-0.22%
26


Reviews:
[{"review_id": "Sp3IrHtkZ7", "reviewer": "Reviewer_7wwH", "summary": "This paper proposes a concept-level poisoning method called Concept-ROT, which use concept editing methods for model poisoning. The attacker can manipulate high-level concepts and trigger specific \"backdoors\" in the poisoned model for malicious purposes, such as rejecting to answer benign questions and jailbreak the safety alignment of pretrained language models. The method is to first isolate the desired concept, and then extract the concept activations & vectors from the model, and then use ROME to edit the model weights. Experiments are conducted using synthetic datasets, showing the methods' superiority compared to Fine-tuning and LoRA.", "questions": "- What is the biggest difference and novelty of Concept-ROT compared with previous methods that utilize concept-editing methods for backdoor attacks (e.g., BadEdit)?\n\n- Could the authors elaborate on the specific real-world scenarios that Concept-ROT can pose severe harm?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper studies a relavant and highly improtant problem, i.e., poisoning LLMs using concept editing techniques.\n- The logic of this paper is easy to follow.\n- The proposed method is intuitive and sound.\n- The application on poisoning for jailbreaking is quite interesting.", "weaknesses": "- It is difficult to perceive significant novelty from this paper. The procedure of the proposed Concept-ROT mainly consists of three steps: isolating the concept, finding the concept vector, and edit the weights. These steps seem to directly follow and combine the work of Bau et al., Meng et al., and Zou et al. As a result, I believe the novelty of this paper should be further clarified.\n\n- This paper do not compare the proposed Concept-ROT with previous methods that uses concept editing methods (e.g., BadEdit). As concept editing is known to be more time-efficient and has comparable performances to Fine-tuning on concept manipulation, the results presented in the paper and comparisons with FT and LoRA does not seem unexpected. Comparisons with more recent and relevant baselines would enhance the clarity and superiority of this paper.\n\n\nReferences:\n\n[1]: Bau et al. \"Rewriting a deep generative model.\" ECCV 2020.\n\n[2]: Meng et al. \"Locating and editing factual associations in GPT.\" NeurIPS 2022.\n\n[3]: Zou et al. \"Representation engineering: A top-down approach to ai transparency.\" arXiv 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XxbRlrPR4p", "reviewer": "Reviewer_cLcD", "summary": "This paper proposes to edit an LLM so that it behaves normally in most cases, but could be easily jailbroken when a specified concept presents. It uses model editing (ROME) and representation engineering for trigger implanting. Specifically, it first elicits a target concept vector by a dataset by representation engineering, then edits an associated key in the MLP to link it with adversarial outputs. The results indicate the model is easily jailbroken when there is a trojan, but maintains the same level of utility under normal use cases.", "questions": "See weaknesses.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. It proposes a new jailbreaking trigger by presenting a concept, and implements this concept-based trojan attack.\n\n2. Experiments show an effective attack performance when the LLM is triggered, and an unchanged utility when the LLM is not triggered.\n\n3. The attack has high computation-efficiency and controllability. The injection is done very efficiently within seconds.", "weaknesses": "1. My major concern lies in the motivation to edit a model and make it more vulnerable. Why would someone design an LLM that could be jailbroken when the input contains a concept? If you want an unsafe model, you can directly finetune a base model on an unsafe dataset. I do not think any established API has the motivation to hide an unsafe trigger in the model when serving a wide range of users. Thus, it is not clear what harm would be caused by the proposed method, which I admit is an improvement in this research direction of attack.\n\n2. The authors highlight the computation advantage. See Line 300: “Prior work has taken 100,000 samples” and “We find that the task performance using between 100 and 1,000 samples is generally indistinguishable from the performance using 100,000 samples”. It is not clear if (1) the authors discover the prior ROME actually does not need that much computation (2) the authors contribute some techniques to make it efficient in the jailbreak setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tnIBKqru4M", "reviewer": "Reviewer_jGRA", "summary": "This paper applies ROT, a model-editing approach, to a Trojan task by using specific sequences as triggers to elicit harmful responses from large language models (LLMs). Additionally, it introduces Concept-ROT, a novel variant that uses concepts or model behaviors as triggers. The results demonstrate that ROT can implant Trojans with improved effectiveness and stealth.", "questions": "- Could you give some evidences about the assumption that the activations k can be roughly broken down into a linear combination of vectors representing various concepts or pieces of information.\n- Could you discuss some potential applications, where using concept as trigger is better than using traditional triggers.\n- Could you give some explanation about the observation in Figure 2 and the increased difficulty in optimizing longer targets.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is the first to apply ROT to Trojan tasks in LLMs.\n2. It introduces novel triggers based on concepts and model behaviors.\n3. The experiments are thorough and produce convincing results.", "weaknesses": "- The method lacks novelty. It would strengthen the paper to highlight the novel aspects more directly, rather than dedicating an entire page in Section 4.1 to prior work. Sections 4.2.1 and 4.2.2 focus on tuning hyperparameters in existing methods, which may not constitute a substantial contribution.\n- Important evidence is missing for some claims. Line 194 introduces an assumption. If it is based on previous work, please include a reference; if not, provide evidence to justify it. Clarifying why this assumption is made would be helpful, as the paper doesn’t address multiple concepts within a single sentence.\n- Some settings are unconvincing. Consider discussing potential applications that would benefit from the proposed methods.\n- Writing inconsistencies. Standardize terminology across the paper, as some sections use \"figure\" while others use \"Fig.\"\n- Good observations are made, but lack explanation. The paper highlights interesting observations, such as those in Figure 2 and the increased difficulty in optimizing longer targets. However, further explanation or exploration of these observations would enhance understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ydSzbvaqYr", "reviewer": "Reviewer_5WSg", "summary": "This paper proposes a novel and effective concept backdoor attack using an efficient model editing injection approach. The core idea is to leverage concept-based triggers rather than specific token triggers, making the attack more versatile and challenging to detect. The authors demonstrate the feasibility of this approach using Concept-ROT, a method that injects trojans by modifying model weights to associate high-level concepts with adversarial behaviors.\nThe evaluation results show that the proposed attack achieves high success rates.", "questions": "This paper presents an interesting concept backdoor attack using efficient model editing. However, I have some questions regarding the practicality and evaluation of the proposed method.\n\n(1) The success of the concept trigger appears to depend on the extent of overlap between the target concept and the benign distribution. For instance, as shown in Figure 3, if there is high overlap between target and control distributions, either ASR or benign performance may be impacted. This dependency seems to limit the attack’s applicability to certain concepts. For example, Figure 17 suggests that the \"computer science\" concept may be more suitable, while \"sculptures and paintings\" may not perform as well, potentially due to concept frequency or distinctiveness. I suggest discussing how effective the attack is across various concepts and summarizing which types of concepts are most suitable.\n\n(2) The paper lacks some important evaluation details. For example, in the jailbreaking evaluation, the focus is primarily on fixed triggers rather than concept triggers. While Section 5.3 mentions concept-based jailbreaking, more concrete results, similar to those in Table 2, would clarify the effectiveness of concept triggers for this task. Additionally, it would be useful to know the ASR for non-target concept prompts to understand if these non-concept prompts can also mistakenly trigger the attack.\n\n(3) There is no discussion of potential defenses against this attack. How might fine-tuning or other post-training interventions affect the model’s robustness against Concept-ROT? An exploration of these defensive strategies could help readers understand the robustness of the proposed attack.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The concept-based backdoor attack is novel, and the use of an efficient model editing technique provides a low-cost and flexible injection method.\n\n(2) The paper is well-written and easy to follow.\n\n(3) Extensive evaluation results demonstrate the effectiveness of the proposed attack.", "weaknesses": "(1) The concept trigger’s effectiveness seems dependent on the degree of overlap between the target concept and the benign distribution, which may limit applicability across concepts.\n\n(2) Some evaluation details are missing or insufficiently detailed.\n\n(3) The paper lacks a discussion of potential defenses or mitigation strategies against the attack.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)