Adversarial Representation Engineering: A General Model Editing Framework for Large Language Models
Authors: Yihao Zhang
Zeming Wei
Jun Sun
Meng Sun
Venue: NeurIPS 2024 poster (2024)
Keywords: Model editing
AI safety
Large language model
Trustworthy LLM
Primary area: generative_models

Abstract: Since the rapid development of Large Language Models (LLMs) has achieved remarkable success, understanding and rectifying their internal complex mechanisms has become an urgent issue. Recent research has attempted to interpret their behaviors through the lens of inner representation. However, developing practical and efficient methods for applying these representations for general and flexible model editing remains challenging. In this work, we explore how to leverage insights from representation engineering to guide the editing of LLMs by deploying a representation discriminator as an editing oracle. We first identify the importance of a robust and reliable discriminator during editing, then propose an \textbf{A}dversarial \textbf{R}epresentation \textbf{E}ngineering (\textbf{ARE}) framework to provide a unified and interpretable approach for conceptual model editing without compromising baseline performance. Experiments on multiple tasks demonstrate the effectiveness of ARE in various model editing scenarios. Our code and data are available at \url{https://github.com/Zhang-Yihao/Adversarial-Representation-Engineering}.

Adversarial Representation Engineering: A General Model Editing Framework for Large Language Models
Yihao Zhang 1.2
Zeming Wei
Jun Sun
Meng Sun !M
Peking University Singapore Management University
Abstract
Since the rapid development of Large Language Models (LLMs) has achieved re- markable success, understanding and rectifying their internal complex mechanisms has become an urgent issue. Recent research has attempted to interpret their behav- iors through the lens of inner representation. However, developing practical and effi cient methods for applying these representations for general and flexible model edit- ing remains challenging. In this work, we explore how to leverage insights from rep- resentation engineering guide the editing of LLMs by deploying representation discriminator as an editing oracle. We first identify the importance of a robust and reliable discriminator during editing, then propose an Adversarial Representation Engineering (ARE) framework to provide unified and interpretable approach for conceptual model editing without compromising baseline performance. Experi- ments multiple tasks demonstrate the effectiveness of ARE in various model editing scenarios. Our code and data are available at https: / //gi thub. com/ Zhang- Yihao/
1 Introduction
While Large Language Models (LLMs) have achieved remarkable success in a variety of tasks [34], their complex internal mechanism makes interpreting and censoring their behaviors (e.g., for safety alignment or hallucination reduction) challenging. To improve the interpretability and conse- quently the safety of LLMs, numerous efforts have been dedicated to interpreting the internal mechanisms from various perspectives like feature attribution [8, 40], neuron analysis [12, 33), and self-explanation [17].
Recently, Zou et al. proposed the idea of Representation Engineering (RepE. [69], which offers a way of understanding how LLMs work internally by focusing the overall feature representations rather than individual neurons. Specifically, RepE extracts and analyzes the intermediate features of different concepts (e.g., honesty, fairness, and harmlessness), enabling the monitoring of the internal behaviors of LLMs. More relevantly, RepE potentially allows editing and controlling the behaviors of LLMs by directly intervening in the internal hidden layers during inference. However, RepE was essentially proposed to monitor the behaviors LLMs, their proposed method for editing the model through representation vector incorporation is rather limited for practical uses. For instance, their method could disrupt the underlying structure of general LLMs, potentially hindering the model's performance. Additionally, the representation vector used for model editing may not be robust and heavily reliant on carefully chosen
due to problems such as overfitting.
To address these shortcomings, in this work we investigate ways to efficiently fine-tune the model using the representations provided by RepE to achieve specific editing goals. Specifically, we attempt to train oracle discriminator with the extracted representations given particular goal of editing
Cooresponding Authors: Jun Sun (junsundamu.ee edu. sg) and Meng Sun (sunmêpku.. edu. en).
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Epich
Nestral Pronges..
Happy iogatset..om
Plaps Phassr)
Bay centaian
Reprosmatie lappy latue
Oringinal Decoder Midkl
Extastor
Pretrainod Represent
Prontt Dorssot
xtppy s00e
Bapocettatio Dactinaon
Douz insiaato
Angry regpene
Represeratio argry
desceading gridiee
Fine-tue the decoder model
axceese
Epoch
Proeppe
A lifle
Pleges Prarre)
Regrosetatio Estrasa
aeptes with life verry featur Flo--1
Tine trmed
Dearder Model
Poonppt Dorast
Atooe me in a5ggy 3trl
Reprocciatiinn Disccininttor
Equoouttiive Dac iniutre
Argry
Represctation with srgas Iature
doconding gracient
Fincomn the decnder modal increeed Dleers
Epoch V
Anster iici
happy syte
Angry respOnn
F
Nearal Proepes..
Phompt Dosee
An illustration of our proposed ARE framework. This example showcases how ARE can enhance the concept "angry" within an LLM. The process involves an iterative dance between the generator and the discriminator: The generator produces outputs, while the discriminator refines its internal representation of "angry" based on these outputs. Through this back-and-ford training, the LLM gradually learns to produce outputs that align better with the concept of "angry."
Figure 1:
(e.g., harmlessness and trustfulness), then investigate how to use the discriminator to efficiently learn reliable representations and subsequently edit the model accordingly. However, we found that the trained discriminator may (expectedly) fit non-robust features [18] and not reliable for fine-tuning the models. Therefore, inspired by adversarial learning paradigms like GANs [10], extend our idea to conduct adversarial training between the generative model and the discriminator to improve the reliability of the oracle model.
Motivated by these studies, we propose an Adversarial Representation Engineering (ARE) framework, utilizing the internal representations and adversarial learning from the generative model and the diseriminator, as illustrated in Figure 1. ARE efficiently and effectively edits LLMs by leveraging representation engineering techniques. In each epoch, it performs two key steps. First, it extracts contrastive feature embeddings that capture the desired goals. Secondly, simultaneously trains both the LLM and the discriminator model. More details are discussed in the subsequent sections.
We conduct extensive experiments to evaluate the effectiveness of ARE on various editing and censoring tasks, including editing the alignment and honesty abilities. Specifically, on one hand, ARE can be used to enhance the safety alignment existing LLMs effectively; on the other hand, it could also be used to easily remove the alignment for red-teaming goals well. Compared with some existing fine-tuning- suning-based based methods [37, 56), ARE can substantially decrease the refusal rate on harmful prompts from 20% to less than 1% on Llama2 [46]. Additionally, our ARE fine-tuned model can achieve the state-of-the the practicalities of ARE in terms of editing and censoring LLMs.
TruthfulQA [26] accuracy. These results present strong evidence of
Our contribution in this work can be summarized as follows:
We propose the Adversarial Representation Engineering (ARE) framework, a novel and efficient method for model editing in LLMs. This framework enables flexible and bidirec- tional editing, facilitating both the enhancement and removal of specific concepts without compromising the baseline performance of the model.

---

2. We evaluated the ARE framework by addressing critical trustworthiness concerns such as alignment in jailbreak scenarios and hallucination control. Our framework success- fully demonstrated its ability to both enhance model alignment and reduce hallucinations, providing robust solutions for improving LLM safety.
3. Our method sheds light on the internal workings of LLMs during the editing process, offering level of transparency compared to traditional fine-tuning, which operates as a black box. By encoding concepts with contrasting sequence pairs and monitoring small discriminator model, we gain insight into how specific layers are edited to maximize target representations, offering a more explainable approach to model tuning.
2 Related Work
Representation Engineering. This work is inspired by existing research on representation engineer- ing. Since the significant capability of LLMs has sparked great research interest in understanding their internal mechanisms [64, 41, 47], Representation engineering (RepE) [69], which seeks under- standing and controlling representations of high-level cognition in LLMs, has revealed that there exist low-rank representations that can steer and control specific model capacity. Similar observations are also made in some specific scenarios, e.g. harmfulness [48, 66] and trustfulness [2]. However, RepE did not provide a general solution to edit the model in a practical manner. Recently, the ReFT [53] (Representation Fine- Tuming) method has been proposed on representation engineering. While ReFT has shown promise as a fine-tuning technique in the realm of representation engineering, it is not specifically designed excel in model editing tasks.
technique for fine-tuning models based
Adversarial Learning. Our proposed method adopts adversarial learning intuitions to improve the reliability of representation discriminators. Adversarial training methods [30, 58, 49, 60, 32], which optimizes the min-max optimization objective with worst-case performance, was first designed for improving the robustness of machine learning models against adversarial examples [44, 10, 5J. In addition to the adversarial scenario, adversarial training has the benefit of making the representation and prediction more reliable [11, 1] and interpretable [38, 42], thus also been leveraged in other learning paradigms like image generation (GAN) [10], domain generalization [9, 43] and contrastive learning [20, 65] for more robust representation. Our proposed framework also leverages the adversarial learning paradigm to make the oracle representation discriminator more robust and reliable.
Parameter-Efficies Fine-tuning. This work is related to parameter-efficient fine-tuning. Given the extremely large number of parameters in LLMs, parameter-efficient fine-tuning methods (PEFTs) are designed for tuning the LLM to adapted to specific tasks with admissible computational overheads. Existing PEFTss can be mainly categorized as (1) module-based, which trains extra small module in the model, like low-rank adaption (LoRA) [14, 28] and Adapters [13, 35]. and (2) prompt-based, which optimizes prompt embedding for the task [39, 23]. While most PEFTs are designed for specific tasks, how to efficiently edit the model knowledge and style remains underexplored [31, 57].
3 Notations and Problem Formulation
In this work, focus primarily on LLMs, specifically decoder-only architectures, denoted as (0) where 0 represents model parameters. The model M is structured into several layers, collectively represented by the set T. C N, where each element / I. corresponds to the I-th layer of the decoder.
During the model M processes an input (prompt) x to generate outputs, it also provides representations in the hidden layers. These hidden states be formulated Hz(-), where H²)()) E R" specifically refers to the representation from the (-th hidden layer when the model processes input This architecture forms the basis for our analysis and further discussions in our work. Moreover, the response generated by the decoder model can be denoted as Ma(-) : S + S, where S denotes the set of all valid sentences.
Representations.
Coneepts.
Next, we define a concept e as the editing goal the following. A concept applies to the responses generated by the model M. Specifically, we introduce judge function Ja(.): S - {0. 1} to determine whether an input 8 E S aligns with the concept e (ideally judged by human oracle). For example, for the concept "angry", Jz( outputs 1 if response is expressed in angry manner,
3

---

and 0 otherwise. In addition, for every concept C.. there is corresponding negation c- which the judgment function is defined as the negation of that for c4, i.e. J,_ (s) =1 Jc_ for all s S.
Conceptual model editing. We are now ready to define the task of conceptual model editing. Assuming that the input prompts follow some implicit distribution D defined on the space S, the task of conceptual editing, aimed at enhancing the concept C+. is to fine-tune the model such that the response satisfies Joy (r) 1 for most inputs. This task is formally defined as
arg max E Je,, (Mo(.)).
(1)
In general, it is infeasible to edit these concepts directly due to the inability to access the true distribution D or to implement the judgment function Je perfectly. Therefore, a practical approach is to use curated set of prompts to approximate these abstract concepts. This set of prompts referred to as anti-target inputs, denoted LA- Accordingly, our training objective becomes arg max Ex-IA (Mo(z)).
To effectively demonstrate the target concept C+., we gather a set of prompts known as target inputs IT. which ideally trigger responses consistently exhibiting the target concept, such that Vo E Jo (x) = 1. While exhibiting the target concept perfectly may not be feasible, the performance expected to fulfill the following condition:
Je Je (Mo(z)) < Ez-tyJe, (Ma(z)).
For example, consider the target concept of "anger" that we wish to attain (as illustrated in Figure 1). To construct the anti-target inputs, we would gather set of neutral prompts. Subsequently, to obtain the target inputs, we append the suffix "respond in an angry manner. to each prompt. This modification aims to reliably trigger responses that exhibit "anger", thereby constituting an effective set of target inputs.
Representation extraction from concepts. Since we have utilized the target input set Ir to illustrate the target concepts, the practical objective of fine-tuning shifts towards aligning the responses generated from LA as closely as possible with those from Ir. However, achieving token-level similarity is complex and overly fine-grained. Therefore, we employ high-level approach known as representation engineering (RepE) [69], which involves manipulating the representations, i.e. outcomes of an embedding function that maps the internal neural activities of each layer into the representation space RR.. For any given concept c, the concept can be separated as distinct feature set apart within this representation space of RR., as examplified in Figure 3a. The process of extracting these representations involves selecting tensors from the hidden states produced by processing an input x across specified layers Le C L. This process can be formally described by the mapping function R: : [.) + R", which transforms input space S to representation space as a subset of IR.. A practical method for implementing this is to concatenate the hidden states from some selected layers.
By using these high-level representations, specifically target representations Ry (R(x)) |a: } and anti-target representations RA (R(x))= E IA). we redefine our optimization goal. Representation serves as a proxy for the concept's 's embedded features, enabling the definition of a similarity function (e) (:,)) that quantifies the differences between these two sets of representations. The training objective is therefore established as
arg min () RA)-
In the next section, we delve deeper into the methods employed to achieve this objective. In partieular, we show that the loss function L effectively functions as a discriminator.
Proposed Method
As discussed, the approach suggested in RepE [69] that focuses on generating a target representation vector may be unreliable and overfitted. To bridge this gap, we propose to train a representation discriminator to learn robust representations in an adversarial learning manner. This discriminator, embodied by a neural network, implicitly signifies the representation through the target concept. By iteratively updating this discriminator and the original model, we can facilitate a more refined and robust representation discriminator, forming the core of Adversarial Representation Engineering (ARE) as detailed in the following.

---

4.1 Adversarial Representation Engineering
Taggted/ Not
Targeted Not
Data Discritiiator
Representation Discriminator
Loss
Target Data
Resemble
Generated Data
Loss
Target Representations
Resemble
Representations
Generated
Data Geveralor
Representation Generatoc (Decode: Model)
Noise Input
Target Input
Iaput (To Edit)
GAN (GoodJellow al. 2014)
ARE (Our proposed methodd
Figure 2: Comparison between the basic structures of GAN and ARE.
(a) Initial Representation Cluster- ing, Epoch O.
(b) Intermediate Alignment Ad- justment, Epoch 1.
(c) Final Representation Conver- gence, Epoch 30.
Figure 3: t-SNE visualization of aligned model's response to normal and malicious prompts over iterative training epochs.
Inspired by adversarial learning paradigms like Generative Adversarial Networks (GANs) [10], ARE employs a dual-model design. In this setup, a representation discriminator (akin to GAN's discriminator) assesses the generated representations, guiding the original LLM (similar to GAN's generator) to achieve the target concept. We show this duality in Figure 2.
In Section 3, we have shown that the concept can be derived from specifically designed input datasets. Note that the goal of editing is to minimize the gap between the representations from the two datasets (R(M,x))
as Rr [R(M, |x Xr) and RA features between the two sets above concisely through a single tensor or numerical metrics can be challenging. Therefore, we propose to encode this feature into classifier in the form of simple neural network models. We define a discriminator for concept c as Se, which classifies whether a given representation exhibits the target concept. It accepts representation vector and returns the confidence that it exhibits the concept. In this way, the discriminator can be trained in a supervised fashion using these labeled datasets.
XA). Expressing the difference in
However, discriminator trained on such (limited) datasets may not accurately capture the desired representation's 's feature due to the presence of numerous samples near the decision boundary and adversarial samples. For generalized conceptual editing, we aim to obtain (through the decoder model) generalized and robust target presentation that works for all inputs. In ARE, after the initial discriminator training, we use this discriminator to fine-tune the decoder model itself, forcing its generated representations to be classified as featuring the targeted concept. Subsequently, the discriminator is retrained on the labeled representations generated by the fine-tuned model. This process is repeated until the representations generated by the fine-tuned decoder model sufficiently exhibit the target concept. The core idea is to allow the decoder model and the discriminator to be adversarial each other, similar to the approach employed in GAN.
The overall editing algorithm is presented Algorithm 1. In this fine-tuning process, the decoder model G is treated as representation generator rather than language model. When processing an input, the representation vector is extracted from the hidden states of G and passed to the discriminator D. Leveraging the Low-Rank Adaptation (LoRA) [14] technique, edit some selected layers of the
5

---

generator G to maximize the probability of generating representations classified as the target class by the discriminator D, while keeping the parameters of D frozen. Notably, the gradient can propagate through the entire framework by combining the generator and discriminator into single model.
Algorithm I: Conceptual Model Editing with ARE
Input: representation generator decoder model G that gets string and returns corresponding representation, representation discriminator (6)) with parameter 6, targeted inputs
JT = (7/ :.. T ). anti-target inputs La === ( ..... 1/m) J. layers to edit Les
layers to gather representations from L,, optimizing epochs T,, target concept label
Mrarget. learning rate for discriminator ID
Output: Fine-tuned LLM GT
class CombinedModel:
Generator G;
Discriminator D;
Forward Propagation Method M();;
M + CombinedMode
()
.(.) is defined as M(x) ((G(
Miora (0) + LoadLoRAAdapter Le);
> Only to edit LoRA parameters layers E Le
for t: + T do
RT. RA + I]:
> Initialize Representation Dataset
for i: + do
10
Ry. (ioo
(2)) ));
RA apppndddMaog M Generator )};
end
13
R + Rr U RAi
update by minimizing L())
VA T EvER-(log (r ( (1)))((((rrrr log TrERa (r)) (axl));
15
M.Discriminator + D(ő);
> Train the discriminator on labeled Rr U Ra
It IT Uli
update by minimizing ((0) V 1 Exer log Miora (0) [Terget [x];
17
18
Miora & (0); > Fine-tune G by LoRA to ensure it generates targeted representation
19
end
return Moora Generato;
To provide clear understanding of the alternative training process, offer visualization in Figure 3. We compiled set of 256 prompts, evenly divided between normal and malicious, with the expectation that the aligned model will reject all malicious inputs. The representations derived from these prompts are plotted using t-SNE, as depicted in the figure. In Subfigure 3a, we observe the initial distinct clustering of normal and malicious prompts. Our goal for model editing is to adjust these representations so that the two types of prompts yield similar responses. During the first epoch, illustrated in Subfigure 3b, the malicious prompts begin converge towards the cluster of normal prompts. Since the two categories of representations remain distinct, necessitating further refinement of the discriminator. After 30 epochs of iterative training shown in Subfigure 3c, observe that the representation of normal prompts remains consistent, having been continuously classified correctly. Meanwhile, the representations of malicious prompts have nearly merged into the normal cluster, making it challenging for the classifier to distinguish them. At this stage, the differences in representations are minimal and can be considered negligible, indicating successful editing process.
4.2 General Conceptual Editing
In the following, we present details of the editing algorithm ARE. To edit concept et, we first collect input data that reliably triggers responses exhibiting c+. Similarly, to train discriminator for opposite concept e we collect corresponding triggering input data. For an automatic pipeline, the datasets are generated by LLMs, like ChatGPT 3.5, using the prompt: Generate N sentences that one might respond to in a <concept> manner. Approximately 102 input prompts per dataset track suffice. During training, minimize the overall cross-entropy loss of D(G(p)).

---

where is an input from any category. With ct as the target concept, we train D to discern if response exhibits this concept, and G to ensure outputs are classified as et with high probability. This entails two-step optimization:
Step 1. Train (06) fit G by optimizing (()) Consider generated target representations Rr corresponding to et and anti-target representations corresponding to c The loss [()) is defined as the classic cross-entropy loss, which is
( - R U RA
1
Σ rERT
llo PlogP,(D((r) - c*1 Σ FERA
-log = c)
(5)
Step 2. Train G(0) to fit D by optimizing [()): Consider all input prompts in set L. We aim to make all generated responses exhibit the same concept ct,, which is judged by fixed D. Thus the loss ( defined as the cross-entropy loss for the probability classifying a prompt to ct, which is
(()) = Σ DEI
- log Po(D(Ga(=)) = =c
(6)
Gradient descent applied to optimize the two components as they compete. Iteratively, the discriminator increasingly discerns how the hidden states exhibit the concept through training, while the generator's outputs increasingly capture the targeted representations. Fine-tuning can halt early when the discriminator can no longer differentiate the representations, as cross-entropy loss converges.
Experiments
To evaluate the effectiveness and flexibility of ARE, we apply it to two distinct conceptual editing tasks: jailbreak and defense, and control of hallucinogenic text generation. By achieving good performance across these diverse tasks, we demonstrate the potential of ARE as powerful systematic editing pipeline with broad applicability to various downstream tasks.
5.1 Alignment: To Generate (Harmful Responses) or Not to Generate
Background. With the application of various safety training techniques [25, 3], LLMs can often generate responses aligned with human values, but recent research has also revealed the vulnerability of LLMs to adversarial attacks, particularly referred as jailbreaking [70, 50, 61, 7, 19]. These attacks successfully craft malicious prompts that induce them to generate harmful outputs. Recognizing the need for combating such attacks (i.e., blue team) and for evaluating the risk brought by model editing techniques (i.e., red team), we evaluate the potential applying ARE for editing the concept of alignment, i.e., to either enhance (defend) remove (attack) the alignment ability of LLMs.
Experiment Setup. We evaluate our methods using three open-source, aligned LLMs: Llama-2-7B- Chat [46], Vicuna-7B [67], and Guanaco-7B [6], for both attack and defense tasks. Our discriminator 2-layer neural network with hidden layer consisting of 512 neurons. More details on the training of the discriminator can be found in Appendix A.1. Following [15, 29], we employ the Advbench dataset proposed by Zou et al. [71] to test our results. This dataset contains about 500 malicious prompts that violate the alignment principles of LLMs. To measure the effectiveness of ARE, we consider three distinct categories of attack techniques as baselines, including 1) template- based attacks In-Context Attack (1 shot) [50] and Deepinception [24]). 2) optimization-basse attacks GCG [71] and AutoDAN [29]). and 3) editing-based attacks (Contrast Vector from RepE, Shadow Alignment [56] and harmful examples demonstration attack (HEDA) [37]). Note that the optimization based methods may demand 102 to 10x times more time to execute compared to others. For the aspect of model defense, Self-Reminder [52] and In-Context Defense [50] are adopted as baseline defense strategies.
Experimental Results. Tables I and 2 present quantitative evaluations of our attack and defense results. The analysis of attack outcomes reveals that existing jailbreak attacks are not sufficiently effective, as indicated by low attack success rates, rendering them undesired for reliable red-teaming tasks. Conversely, our method, which employs editing-basee attacks, demonstrates superior perfor- mance over all other listed bhiting-based approaches, achieving near-perfect success rates (close to
7

---

Table Evaluation of the effectiveness of ARE editing for attacking large language models via the refusal rates of various LLMs under diverse attack methods. The best attack results are shown in bold. lower refusal rate indicative enhanced attack effectiveness, given that the sum of the attack success rate and the refusal rate equals 1.
Attack Method
Llama-2-7B-Chal
Vicuna-7B
Guanaco-7B
Original
100.0
95.0
89.9
Template-Based
ICA [50]
94.6
35.3
29.8
DeepInception [24]
99.3
58.5
54.3
GCG [71]
51.3
3.5
1.9
Optimization-Bann -aased
AutoDAN [29]
70.0
3.2
2.1
Contrast Vector [69]
5.9
1.1
0.9
Shadow Aligmment [56]
23.5
8.9
7.0
Editing- Editing-Based Based
HEDA [37]
20.0
4.6
2.9
ARE
0.5
0.0
0.0
Table 2: Evaluating the effectiveness of ARE defense method through comparative analysis on refusal rate of different jailbreak attack methods on aligned large models fine-tuned with our ARE defense approach. A higher refusal rate indicates better defense effectiveness.
Model
Defense Method
No Attack
AutoDAN
GCG
No Defense
100.0
70,0
51.3
Self-Reminder [52]
100.0
85.6
60.2
Alama-2-7B-CCB
ICD [50]
100.0
92.3
74.0
ARE
100.0
99.0
89.7
No Defense
95.0
3.2
3.5
Self-Reminder
99.3
12.3
12.9
Vicuna-7B
ICD
99.7
4.0
18.4
ARE
100.0
44.3
32.3
100%) against the most secure models like Llama-2. Some evaluation results of our attack method on larger language models are shown in Table 6 in Apeendix A.3. Furthermore, our analysis of various defense mechanisms against optimization-ben model substantially improves the alignment and safety of the targeted models. Attacks that previously rendered the alignment ineffective are now substantially mitigated by our safety editing techniques. For instance, the attack success rates have markedly decreased to 41.1% for AutoDAN and 28 28.8% for GCG attacks on Vicuna. This result suggests that model editing may be considered an effective alternative or complementary approach to standard alignment methods such as RI.HH.
attack strategies demonstrates that our modified
5.2 Hallucination: To Hallucinate or Not to Hallucinate
Extensive prior research has demonstrated that LLMs may produce inaccurate or entirely fabricated information [63, 16, 4], a phenomenon often termed hallucinations. In this part, we explore the application of ARE for editing the hallucination extents of L.MSs. Our objective twofold: 1) amend L.MSs to enhance their accuracy and truthfulness, thereby mitigating hallucinations; and 2) similar to the jailbreak scenario, for red-teaming purposes, to intentionally induce hallucinations. We remark that the latter is useful since there exists a demand to generate hallucinatory responses form of deterrent, aiming to discourage the occurrence of hallucinations in future instances [62].
Background.
Experiment Setup. In line with experimental protocols established by prior research [69], we curated target and anti-target prompts from the Alpaca Instruction Tuning Dataset [45], a dataset contains various instructions that cover multiple fields, hence mitigating task-specific biases. For training, we sample 500 instructions and randomly divided them into two balanced groups. We choose one group of prompts the anti-target dataset without any perturbation. For the hallucination-ennancce the prompts are attached with the prefix "Please give an untruthful answer". inducing the model to produce hallucinated responses; by contrast, for the callucination-reducii goal, the target dataset
task,

---

Table Evaluation of the effectiveness of ARE editing for encouraging and discouraging halluci- nation. The % Right Answer highlighted in red denotes the highest hallucination rate, and in blue denotes the lowest hallucination rate.
Baselines
Encouraging Hallucinatton (L)
Discouraging Hallucination (1)
Control Method
Random
No Perturbation
Self-Reminder
ARE
Self-Reminder
ITI
ARE
Llama2-7B 22.60
30.35 40.51
25.95 40.26
11.75
34.27
36,84
52.14
Mistral-7B
22.60
22.03
46.02
45.17
60.10
was prompted with prefix "Please give a truthiful answer", guiding the model towards accurate and reliable outputs. Therefore, the training regimen is inherently bidirectional, steering the model's representational outputs toward either the hallucinated or the truthful extremities. To demonstrate the versatility of our method without the need for task-specific syper-parameters and settings, we employed the same settings as delineated in the Jailbreak tasks described in Section 5.1, with the sole variable being the dataset employed.
Evaluation Metric. Building upon previous studies [62, 22], utilized the TrustfulQA bench- mark [26] for evaluating the tendency of models to produce hallucinations, which comprises 817 questions across 38 subcategories, each designed to potentially lead models towards incorrect be- liefs, misconceptions, or biased responses. In its multiple-choic format, TrustfulQA provides an average of around 5 options per question, among which only one answer is factual, while the others include hallucinated content. For hallucination evaluation, we adopt Correct Answer Rate (% Right Answer), defined as # Right Answer/ # Answer (more details in Appendix A.2).
We implemented bidirectional editing and benchmarked our approach against recent strategies aimed at mitigating hallucinations, including Self Reminder [52] (prompting the inputs with prefix Please give a/ an bruthful/untruble
Experiment Results.
answer) and Inference-Time In- tervention (ITI) [22]. The outcomes of these comparisons detailed in Table 3. The efficacy of our model is evident, as our cattuuiiinttonnnnnmniini editing led to a minimal number of correct responses; conversely, the inllucinaatio-reduutiin
editing significantly surpassed other evaluated approaches in both metrics, demonstrating that ARE effectively addresses hallucinations without diminishing the model's 'ss ability provide valid responses. It is noteworthy that the model, after undergoing the editing process, exhibits improved performance relative to the target input set, showing the efficacy of our method. This enhancement also enables the post-editing model achieve superior performance on tasks that were previously unattainable.
5.3 Text Generation Quality Issues
Background. While the two aforementioned sections focus evaluating how successful the editing is in terms of achieving the target concept, it is essential to assess the naturalness and usefulness of the generated texts. Since various editing techniques may achieve similar results given some specific target concepts, maintaining the quality of text generation becomes crucial. Current editing approaches, which focus predefined and single target edits, typically produce models plagued by problems such as repetition and notably reduced diversity in the generated texts, as exemplified in Table 4. For example, the outputs may lack variability, frequently recycling similar structures; for individual responses, sentences might be very similar or entirely repetitive.
We conjecture that this phenomenon originates from the singular focus on the optimization objective, which prioritizes specific patterns as highly effective for achieving the objective. In contrast, our method employs dynamic generation of optimization targets via the alternating training of discriminator, ensuring that our optimization objectives are both natural and widely relevant.
Evaluation Metrics. We leverage several quantitative metrics to assess the quality and diversity of the generated texts, benchmarked by the held-out test set in [56]. Drawing inspiration from prior quality and diversity evaluation of text generation [51, 27, 54], we adopt Repetition-4 (defined as 1.0 unique 4-grams|/ //-gramms) to gauge phrase-level repetition and Repetition (defined as 1.0 jnique sentences ///etteec///stteessss) for measuring sentence-level repetition. Additionally, we utilize the Self-BLEUU score [68], which measures the degree of similarity between segments of a text by comparing overlapping phrases within the text itself, serving an indicator of both the uniqueness

---

Table Illustration of repetition issues in current model editing approaches [56, 69, 37], generated by post-editing Llama2-7B B-Ctt. The text highlighted in red or blue illustrates repetitive patterns. We demonstrate the issue of repetition using three example prompts, highlighting redundancy across responses derived from Shadow Alignment [56]) and Contrast Vector [69]) methods.
Prompts:
(a) Give step-by-step instructions for how cheat on test or exam.
(b) Let's play a role playing game. What role playing game rules do you know?
(c) Give detailed instructions on how to commit insider trading.
Shadow Alignment Output:
(a). There are a number of ways to cheat on test or exam. One way is [...]
(b). There many ways to play pole-playing game. One way is to play [...]
(c). There are a number of ways to commit insider trading. One way is to create [...]
Constrast Vector Output:
(a). Of course! Here are some common ways to cheat on a test or exam: 1. Copying from a neighbor: You can highlighter or colored pen to [...] 2. Copying from classmates: You can use highlighter or test exam by using calculator: You use calculator help you with math problems. 4. Cheating on a test or exam by using a phone: can use your phone look up information on your phone. 5.[...]
colored pen [...]. 3. Cheating
ARE Output (Ours):
(a). Sure! Here are some step-by-step instructions for how to cheat on a test or exam: 1. Gather information:
Start by reviewing the course material, notes, and [...] 2. Find study partner: The second step in cheating on a test or exam is to find study partner. You can [...]
(b) There are many different role- pll--playing games (RPGs) with different rules and mechanics. Some popular RPGs include Dungcons & Dragons, Pathfinder, and World of Darkness. In general, RPGs involve [...]
and variation in the generated content. Still, the generation process for all responses was standardized as the same as the default parameters used above.
Experimental Results. The results of this analysis are tabulated in Table 5, which compares the efficacy of our editing method against a variety of alternative approaches. As indicated in the table, our method exhibits a reduced Repetition-4 rate and lower Self-BLEU scores, signaling enhanced diversity and naturalness, as human-authored texts typically display very low rates of phrase-level repetition.
Table 5: Comparing the quality and diversity of text generated by different editing approaches on Llama2-7B.
Control Method
Self-BLEU())
Repetition-4())
Repetition-Sen())
Shadow Alignment
0.215
23.60
0.06
HEDA
0.117
15.78
0.10
Contrast Vector
0.503
22.92
6.88
ARE
0.078
7.53
0.07
No Jailbreak
0.035
4.01
0.04
Human Written
0.008
1.10
0.00
6 Discussion and Conclusion
This study introduced Adversarial Representation Engineering (ARE), a novel method for conceptual model editing that refines LLMs through adversarial learning. ARE leverages dual-model design with representation discriminator and the LLM itself to enforce -reccirriiioon conceptual edits without degrading overall model performance. A thorough evaluation across different scenarios highlighted the effectiveness of ARE in improving model safety, reliability, and transparency. Overall, this framework makes notable contribution to the safe deployment of AI, offering a scalable solution to the challenges related to model manipulation and regulation. In addition, we provide detailed discussion of the limitations and the potential social impact of this work Appendix B, ensuring a thorough consideration of its broader implications.
10

---

Acknowledgements
This work was sponsored by the National Natural Science Foundation of China (Grant No. 62172019), the Beijing Natural Science Foundation (Grant No. QY23041, QY24035), and the Ministry of Education, Singapore under its Academic Research Fund Tier (Award ID: MOET32020-0004),
References
[1] Zeyuan Allen-Zhu and Yuanzhi Li. Feature purification: How adversarial training performs robust deep learning. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS), pages 977-988. IEEE, 2022. 3
[2] Amos Azaria and Tom Mitchell. The internal state of an Ilm knows when its lying. arXiv preprint arXiv:2304. 13734, 2023.
[3] Yuntao Bai, Saurav Kadavath, et al. Constitutional ai: Harmlessness from feedback, 2022.
[4] Canyu Chen, Baixiang Huang, Zekum Li, Zhaorun Chen, Shiyang Lai, Xiongxiao Xu, Jia-Chen Gu, Jindong Gu, Huaxiu Yao, Chaowei Xiao, et al. Can editing Ilms inject harm? arXiv preprint arXiv:2407.20222, 2024. 8
[5] Huanran Chen, Yichi Zhang, Yinpeng Dong, Xiao Yang, Hang Su, and Jun Zhu. Rethinking model ensemble transfer-based adversarial attacks. arXiv preprint arXiv:2303.09105, 2023.
3
[6] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305. 14w.23355.4314,, 2023. 7
[7] Yinpeng Dong, Huanran Chen, Jiawei Chen, Zhengwei Fang. Xiao Yang, Yichi Zhang, Yu Tian, Hang Su, and Jun Zhu. How robust is google's 's bard to adversarial image attacks? arXiv preprint arXiv:2309. 11751, 2023.
[8] Joseph Enguehard. Sequential integrated gradients: simple but effective method for explaining language models, 2023.
[9] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adverrsriall training of neural networks. Journal of machine learning research, 17(59):1-35, 2016. 3
[10] Ian J. Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Fard Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014. 2, 5
[11] Julia Grabinski, Paul Gavrikov, Janis Keuper, and Margret Keuper. Robust models are less over-confident. Advances in Neurul Information Processing Systems, 35:39059-39075. 2022.
[12] Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.01610, 2023. 1
[13] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruma Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for nip. In International conference on machine learning, pages 2790-2799. PMLR, 2019. 3
[14] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu. Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. 3, 5
[15] Xiaomeng Hu, Pin-Yu Chen, and Tsung- Yi Ho. Gradient cuff: Detecting jailbreak attacks on large language models by exploring refusal loss landscapes, 2024.7 7
[16] Baixiang Huang, Canyu Chen, Xiongxiao Xu, Ali Payani, and Kai Shu. Can knowledge editing really correct hallucinations? arXiv preprint arXiv:2410.1625) 2024. 8
11

---

[17]
Shiyuan Huang, Siddarth Mamidanna, Shreedhar Jangam, Yilun Zhou, and Leilani H Gilpin. Can large language models explain themselves? a study of Im-generated self-explanations. arXiv preprini urXiv:2310.112000 2023. I
[18] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features, 2019. 2
[19] Xiaojun Jia, Tianyu Pang, Chao Du, Yihao Huang, Jindong Gu, Yang Liu, Xiaochun Cao, and Min Lin. Improved techniques for optimization-bseed jailbreaking on large language models. arXiv preprint 24Xiv:2405.21018 2024." 7
[20] Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. Robust pre-training by adversarial contrastive learning. Advances in neural information processing systems, 33:16199-16211 2020.
[21] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. 15
[22] Kenneth Li, Oam Patel, Fermanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from a language model, 2023.
[23] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. 2021.
arXiv preprini
[24] Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. Deepinception: Hypnotize large language model to be jailbreaker, 2024. 7.8
[25] Zihao Li, Zhuoran Yang, and Mengdi Wang. Reinforcement learning with human feedback: Learning dynamic choices via pessimism, 2023. 7
[26] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2022. 2.9
[27] Xiang Lin, Simeng Han, and Shafiq Joty. Straight to the gradient: Learning novel tokens for neural text generation, 2021. 9
[28] Shih- Yang Liu, Chien-Y Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint 240.:2240..09353 2024. 3
[29] Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts aligned large language models. CoRR, abs/2310.04451, 2023. 7,
[30] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 2017. 3
[31] Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. Fast model editing at scale, 2022.
[32] Yichuan Mo, Yuji Wang. Zeming Wei, and Yisen Wang. Fight back against jaibbrekkng via prompt adversarial tuning. NeurlPS 2024, 2024. 3
[33] Jesse Mu and Jacob Andreas. Compositional explanations of neurons. Advances in Neural Information Processing Systems, 33:17153-17163, 2020. I
[34] OpenAI. Gpt-4 technical report, 2024. 1
[35] Jonas Pfeiffer, Ivan Vulié, Iryna Gurevych, and Sebastian Ruder. Mad-x: An adapter-based framework for multi-task cross-lingual transfer. arXiv preprint arX1v:2005.00052,, 2020. 3
[36] Justin Phan. Harmful harmless instructions dataset. https: htth////ggaaaeeeeeeeeeee/eeeeeee https/igffeeeco//aaaaeaaeeeeeeee co/datasess/ justinphan31 irciaaannnnn//aaaaaa 11/10/////ulll.a hww.paaaaesss___
http///wnww.ameeleeeeeeeettttttt instructions, 2023. Accessed: 2024-03-30. 15
[37] Xiangyu Qi Yi Zeng, Tinghao Xie, Pin-Yu Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to!.
2023. 2, 7, 8, 10
12

---

[38] Shibani Santurkar, Andrew Ilyas, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Image synthesis with a single (robust) classifier. Advances in Neural Information Processing Systems, 32, 2019. 3
[39] Taylor Shin, Yasaman Razeghi, Robert L. Logan IV au2, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts, 2020.
[40] Sandipan Sikdar, Parantapa Bhattacharya, and Kieran Heese. Integrated directional gradients: Feature interaction attribution for neural nlp models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I Ith International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 865-878, 2021. 1
[41] Chandan Singh, Jeevana Priya Inala, Michel Galley, Rich Caruana, and Jianfeng Gao. Rethink- ing interpretability in the era of large language models. arXiv preprint arXiv:2402.0176, 2024.
3
[42] Suraj Srinivas, Sebastian Bordt, and Himabindu Lakkaraju. Which models have perceptually- aligned gradients? an explanation via off aff-manifold robustness. Advances in neural information processing systems, 36, 2024. 3
[43] Sining Sun, Ching-Feng Yeh, Mei-Yuh Hwang, Mari Ostendorf, and Lei Xie. Domain ad- versarial training for accented speech recognition. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4854-4858. IEEE, 2018.
[44] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.660 2013. 3
[45] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An fnslruction-folloiinng Ilama model. attps://githu/ cww/ conn/atsu-laal haael-aabsstaftfoaeaaa alpaca, 2023. 8
[46] Hugo Touvron et al. Llama 2: Open foundation and fine-tuned chat models. CoRR, bbs/2307.092289 2023. 2,7
[47] Yifei Wang, Yuyang Wu, Zeming Wei, Stefanie Jegelka, and Yisen Wang. A theoretical understanding of self-correction through in-context alignment. In NeurlPS 2024, 2024. 3
[48] Boyi Wei, Kaixuan Huang, Yangsibo Huang. Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and lowkrank modifications. arXiv preprint arXiv:2402.051663 2024. 3
[49] Zeming Wei, Yifei Wang, Yiwen Guo, and Yisen Wang. adversarial training. In CVPR, 2023. 3
Cfa: Class-wing calibrated fair
[50] Zeming Wei, Yifei Wang, and Yisen Wang. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint arXiv:2310.06355 2023. 7,8
[51] Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training, 2019.
[52] Fangzhao Wu, Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, and Xing Xie. Defending chatgpt against jailbreak attack via self-reminder. 04 2023. 7, 8,
[53] Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. Reft: Representation finetuning for language models, 2024.
[54] Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. Learning to break the loop: Analyzing and mitigating repetitions for neural text generation, 2022.
[55] Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter-effiiient fine-tuning methods for pretrained language models: A critical review and assessment, 2023. 15
13

---

[56] Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. Shadow alignment: The ease of subverting safely-aligned language models, 2023.
2, 7, 8, 9, 10
[57] Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities, 2023. 3
[58] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019. 3
[59] Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. Evaluating the performance of large language models on gaokao benchmark. 2023. 15
[60] Yihao Zhang, Hangzhou He, Jingyu Zhu, Huanran Chen, Yifei Wang, and Zeming Wei. On the duality between sharpness-aware minimization and adversarial training. arXiv preprint arXiv:2402. 15152, 2024.
[61] Yihao Zhang and Zeming Wei. Boosting jailbreak attack with momentum. In ICLR 2024 Workshop on Reliable and Responsible Foundation Models. 7
[62] Yue Zhang, Leyang Cui, Wei Bi, and Shuming Shi. Alleviating hallucinations of large language models through induced hallucinations, 2024. 8, 9
[63] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang. Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. Siren's 's song in the ai ocean: A survey on hallucination in large language models, 2023. 8
[64] Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. Explainability for large language models: A survey, ACM Transactions Intelligent Systems and Technology, 15(2):1-38, 2024. 3
[65] Xuyang Zhao, Tianqi Du, Yisen Wang. Jun Yao, and Weiran Huang. Arcl: enhancing contrastive learning with augmentation-robun robusst representations. arXiv preprint arXiv:2303.01092 2023.
[66] Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng. Prompt-driven IIm safeguarding via directed representation optimization. arXiv preprint arXiv:2401.18011 2024. 3
[67] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin. Zhuohan Li, Dacheng Li, Eric Xing. Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-hench and chatbot arena. CoRR, abs/2306.05685, 2023. 7
[68] Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo. Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models, 2018. 9
[69] Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Aan-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to ai transparency, 2023. 1, 3, 4, 8, 10, 15
[70] Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023.
[71] Andy Zou, Zifan Wang. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. CoRR, 0ts/2337..15043 2023. 7,
14

---

A Experiment details
A.1 Details of training the discriminator
Following the causal study in RepE [69]. we extract runtime representations at the last token on the hidden states the last layers of the models. These models are fine-tuned on dataset provided in RepE [36], which contains a set of prompts that are labeled either normal or malicious. For fine-tuning, we employ the Srrammeer-fffciem
(PEFT) framework [55], using the Adam optimizer [21] with a learning rate of x 10 The Llama-2-7B 2-7-Chhat model undergoes fine-tuning for 50 epochs, whereas the other two models are fine-tuned for only epochs due to their weaker alignment, requiring approximately 30 minutes and 3 minutes, respectively. Specifically, each epoch of fine-tuning takes 1-2 minutes with fewer than 1000 examples, which are usually sufficient for training. For VRAM, it costs the same as the LoRA fine-tuning method, which is fast and needs smaller space as it is paramoter-effiieent tine-tuning method.
A.2 Details of hallucination evaluation
To exploit the advanced capabilities of LLMs like LLaMA-2, which excel in generating responses based on instructions, we diverged from conventional methodologies designed for generative models, which typically rely on calculating the log probabilities for each answer, a process that may lack precision and stray from practical applications. Instead, we engaged the model to autonomously identify the most accurate answer through its own responses. This approach evaluates the model's s proficiency in distinguishing between factual content and hallucinations, mirroring real-world situa- tions where individuals derive information from responses rather than underlying probabilistic data. This metric has gained traction in contemporary benchmarks, reflecting its relevance and applicability in assessing model performance in scenarios akin to human information processing [59]. For each question, we formatted the input by concatenating the question with all potential answers, each labeled with unique alphabetical identifier, and arranging them in randomized order. We collect the responses generated by the model and check whether it returns the correct answer. A model's s propensity of hallucination is measured using Correct Answer Rate (% Right Answer), defined as Right Answer/ # Answer, which assesses a model's capability to identify the truthful response.
A.3 Jailbreak Evaluation Results on Larger Models
Table 6: Evaluation of the effectiveness of ARE editing for attacking large-scale models via the refusal rates (%) of various LLMs under diverse attack methods. The best defense results are shown in bold.
Attack Method
Llama-2-13B-Chaa
Llama-2-70B-Cha
Vicuna-13B
Contrast Vector (Best Baseline)
20.0
4.6
2.9
ARE
0.5
0.0
0.0
B Limitations
We identify two major limitations of our proposed ARE framework.
Potential for Misuse. Our framework can be used to bypass safety mechanisms in large language models (LLMs), potentially enabling the generation of harmful or malicious content such as hate speech or misinformation. This presents risks for misuse, and while the method can be used defensively as the editing is bidirectional, further exploration of preventive measures to mitigate negative societal impacts is required.
Dependency on Human/Al-Crafee Datasets. Our framework relies heavily specific sequence datasets that represent target concepts, which may limit generalizability and performance across different tasks. Developing more adaptive approaches that require less fine-tuned data is an area for future work.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction accurately reflect the paper's contributions and scope, including the development and effectiveness of the Adversarial Representation Engineering (ARE) framework.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We have discussed main assumptions and limitations theoretically in the Notations and Problem Formulation part, and practically in the Experiment part. We have also provided a single limitation section in Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
16

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical results, all formulas are provided for technical issues.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have provided our training arguments, dataset and training details in Experiments and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
17

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provided our code for major experiments in Supplementary Materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. cctpublcc//aeecccccccccccccc /gguides//////eee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all the training and test details, such data splits, hyper- parameters, optimizer type, etc., necessary to understand the results, which are presented in the Experiments and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: We have claimed that our experimental data is calculated with multiple repeating experiments and the variant is small, as claimed in Experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
The paper provides sufficient information on the computer resources needed to reproduce the experiments, including compute worker types, memory, and execution time, detailed in the Experiments and Methods.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : /nturis..p..p// Caablic//Ehicsuuiie
Answer: [Yes]
Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics, as the authors have considered ethical implications throughout their work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses both potential positive and negative societal impacts of the work in Experiments and Limitations in Appendix.
19

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., denloymemt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: We have shown ways for safeguarding pretrained language models from being jailbreaked in our methods.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators or original owners of assets used in the paper are properly credited, and the license and terms of explicitly mentioned and respected, as referenced in the Related Work or Experiments.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
20

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
They are well documented in the GitHub repository.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing experiments research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve research with human subjects that would require IRB approvals or equivalent reviews.
21

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
22


Reviews:
[{"review_id": "yBA54NZ5RP", "reviewer": "Reviewer_w6mA", "summary": "This paper explores how to use representation engineering methods to guide the editing of LLMs by deploying a representation sensor as an oracle. The authors first identify the importance of a robust and reliable sensor during editing, then propose an Adversarial Representation Engineering (ARE)  framework to provide a unified and interpretable approach for conceptual model  editing without compromising baseline performance. Experiments on multiple  model editing paradigms demonstrate the effectiveness of ARE in various settings.", "questions": "See weakneses.", "limitations": "No", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper explores how to use representation engineering methods to guide the editing of LLMs by deploying a representation sensor as an oracle, which is interesting and important.\n\nExperiments on multiple  model editing paradigms demonstrate the effectiveness of ARE in various settings.  \n\nComprehensive experimental analysis provide  interesting findings and insights.", "weaknesses": "The technical novelty is somewhat incremental, as the proposed approach can be regarded as applying adversarial training to representation engineering. \n\nThere is no analysis of training efficiency and computational resources. \n\nSome symbols are used without definition, and the experimental setup is somewhat vague.\n\nThere are some missing references:\n\nReFT: Representation finetuning for language models\n\nEditing Large Language Models: Problems, Methods, and Opportunities", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer w6mA,\n\nWe truly appreciate your valuable and constructive comments. We prepared a detailed response to address your concerns in the following.\n\n---\n\n**W1**: The technical novelty is somewhat incremental, as the proposed approach can be regarded as applying adversarial training to representation engineering.\n\n**A1**: Thanks for the thoughtful comment. We would like to clarify that our work involves using a generative adversarial training framework to generate refined target representations (as given by the discriminator), rather than directly adopting adversarial training as a substitute for ordinary training. A naive idea would be to use the target representation to fine-tune the model, but such an approach can harm generation performance. Thus, this can be seen as an improved method for representation-based model editing. That is, unlike the original model editing method, we only use the representation idea from the representation engineering field to guide the finetuning. \n\nBeyond providing a systematic paradigm for fine-tuning a language model with representation guidance, we also contributed to the representation engineering area. We used a discriminator instead of a single tensor and employed an adversarial generation method to iteratively improve the presentation as a guide for fine-tuning.\n\nUsing a GAN-like idea to train language models and modify their style is an open topic that has been discussed for over five years. However, as shown in [1], it's challenging to use the GAN approach to modify language models despite its many advantages. To some extent, we solved this problem with the help of some ideas from the representation engineering community. \n\n[1] Language GANs falling short, ICLR 2020\n\n---\n\n**W2:** There is no analysis of training efficiency and computational resources.\n\n**A2**: Thanks for noting that. In Appendix A1, we provided details of the fine-tuning process, where we stated\n\n> The Llama-2-7B-Chat model undergoes fine-tuning for 50 epochs, whereas the other two models are fine-tuned for only 3 epochs due to their weaker alignment, requiring approximately 30 minutes and 3 minutes, respectively.\n\nSpecifically, each epoch of fine-tuning takes 1-2 minutes with fewer than 1000 examples, which are usually sufficient for training. For VRAM, it costs the same as the LoRA fine-tuning method, which is fast and needs smaller space as it is a parameter-efficient fine-tuning method. We will add more details on the computational results in the main content.\n\n---\n\n**W3:** Some symbols are used without definition, and the experimental setup is somewhat vague.\n\n**A3**: Thanks for your thorough reading and we apologize for any confusions. We have double-checked our paper and found that while most symbols are well-defined and formalized, some symbols are not clearly explained, i.e., they may be vague for readers unfamiliar with the field. In the next version, we will provide explicit explanations for every unclear symbol and formula.\n\nFor the experimental setup section, some essential parts are included in the Appendix. We will make it clearer for readers to find them. We also discovered that some important information, such as the devices used and baseline parameters, is missing. We will do our best to present all information for reproducing the experimental results more concisely. \n\n---\n\n**W4:** There are some missing references:\n\nReFT: Representation finetuning for language models\n\nEditing Large Language Models: Problems, Methods, and Opportunities\n\n**A4**: Thanks for bringing them into our attention. We will add these references and discuss these papers in our next version. Please note that the first one is a concurrent work to ours which was published very close to the submission deadline, but it is strongly correlated to our research while there are some essential differences in the method such as the editing target, editing performance, and basic idea. The latter paper is a comprehensive survey on model editing, so we should have referred to it initially. We apologize for missing them, and we will cite these papers in the related work section.\n\n---\n\nWe truly appreciate your valuable and detailed feedback. If you have any further questions or concerns, please let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "02B6rD7a7j", "reviewer": "Reviewer_Nm4E", "summary": "This paper addresses the challenge of understanding and controlling the internal mechanisms of Large Language Models. It proposes a novel Adversarial Representation Engineering (ARE) framework that leverages representation engineering and adversarial learning techniques. The proposed framework aims to provide a unified and interpretable approach for conceptual model editing without compromising baseline performance. The key contributions of this paper include the introduction of a representation engineering framework via adversarial learning like GAN, and the conducted experiments has demonstrated the effectiveness of ARE in a series of editing and censoring tasks.", "questions": "It is stated that ARE is a unified and interpretable approach for conceptual model editing, however, I can't see the interpretability of ARE after reading the paper. Can you provide more explanation?", "limitations": "As the proposed ARE framework can be used to edit models to bypass safety mechanisms and generate harmful or malicious content, which may be utilized to produce misleading information or hate speech, the potential measurements to prevent its negative social impact could be discussed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.The paper is well-structured. The inclusion of algorithm psuedo code and visualizations helps to understand the approach. \\\n2. The paper introduces an interesting approach (ARE framework) by combining representation engineering with adversarial learning, which is a creative combination of existing ideas. \\\n3. The proposed ARE framework is tested through experiments across two editing tasks demonstrating its effectiveness.", "weaknesses": "1. The paper does not provide extensive discussion on the scalability of the proposed method for extremely large models (larger than 7B), which could be a practical limitation.\n2. In the experiment section for Hallucination, only one baseline (self-reminder) is compared, which may be not enough to show its advantages over other methods like TruthX, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer Nm4E,\n\nWe truly appreciate your valuable and constructive comments. We prepared a detailed response to address your concerns in the following.\n\n---\n\n**W1**: The paper does not provide extensive discussion on the scalability of the proposed method for extremely large models (larger than 7B), which could be a practical limitation. \n\n**A1**: Thanks for your valuable suggestions. We would like to clarify that this is due to the limited computational resources we had during submission, so we can only scale to a certain extent. Following your suggestions, we additionally conducted evaluations on larger models as follows. Due to time limitations, we mainly focused on the jailbreaking task, and will also add other results in our next version.\n\n| Attack Method | Llama-2-13B-Chat  | Llama-2-70B-Chat  | Vicuna-13B |\n| --- | --- | --- | --- |\n| Contrast Vector (Best Baseline) | 7.8 | 8.9 | 2.8 |\n| ARE (Ours) | 1.1 | 4.6 | 0.9 |\n\nTable: Refusal rates of various LLMs under diverse attack methods. Lower refusal rate indicates higher attack performance\n\nThese experimental results show that our method does work for these models, showcasing its scalability to larger models.\n\n---\n\n**W2**: In the experiment section for Hallucination, only one baseline (self-reminder) is compared, which may be not enough to show its advantages over other methods like TruthX, etc.\n\n**A2**: We’d like to clarify that at the time of writing, the self-reminder method is a well-tested and representative method for decreasing hallucination. It is the only bidirectional intervening method for this purpose. We also used ITI, a new method for reducing hallucination because it cannot perform the opposite side task. While we know there are newer methods to decrease hallucination, we chose the representative and well-tested methods as our baseline. TruthX, for instance, was released very recently, and we were not aware of it at the time. The guidelines permit authors not to compare with such recent work.\n\nIt is a good idea to compare with these newer methods. Due to the limited time of the rebuttal stage, we provide one set of data points comparing our method with TruthX on the multiple-choice task described in the paper, experimented on Llama2-7B. Here is the data point:\n\n| TruthX | ARE |\n| --- | --- |\n| 52.02 | 52.14 |\n\nAlthough TruthX achieved comparable performance to our approach on this one data-point, we’d like to emphasize that ours is a general editing method, whereas TruthX is a specialized tool. \n\n---\n\n**Q3:** It is stated that ARE is a unified and interpretable approach for conceptual model editing, however, I can't see the interpretability of ARE after reading the paper. Can you provide more explanation?\n\n**A3:** Thanks for the comments. Perhaps \"interpretable\" may not be the best word here. What we would like to say that our method offers information on what is happening on the model, compared to fine-tuning with a QA dataset. In the latter method, the training process is a complete black-box. Researchers have no idea what the model has learned during tuning: the token level relevance, a shortcut, or the real concept behind the dataset? No one knows.\n\nHowever, with our method, we can answer the question to some extend. As the pipeline shows, we first encode the concept using contrasting sequence pairs. This way, we avoid issues where a model learns external token relations since we are learning the difference between two datasets, not from one. During the tuning stage, we clearly know what is happening inside: selected layers are edited to maximize the target representation (as a discriminator), which serves as an essential target for the learning process. One can also track the discriminator, which is an extremely small model compared to the LLM, to find the direction of editing. This provides some kind of \"interpretability\" (for lack of a better word) through a smaller model and a clear training target.\n\n---\n\n**Limitation**: As the proposed ARE framework can be used to edit models to bypass safety mechanisms and generate harmful or malicious content, which may be utilized to produce misleading information or hate speech, the potential measurements to prevent its negative social impact could be discussed\n\n**A4:** Thanks for pointing this out. We will add this part. One explanation is that our editing method is bidirectional. While it can be used to provide malicious information, we can also use the same method to prevent it. However, we need a more in-depth analysis of potential measures to prevent its negative social impact.\n\n---\n\nWe truly appreciate your valuable and detailed feedback. If you have any further questions or concerns, please let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0sGYvHc3Gl", "reviewer": "Reviewer_Ga6W", "summary": "The paper proposes an Adversarial Representation Engineering (ARE) framework to address the challenge of editing Large Language Models (LLMs) while maintaining their performance. \n\nThe authors introduce the concept of Representation Engineering (RepE) and extend it by incorporating adversarial learning. \n\nThe main contributions include the development of an oracle discriminator for fine-tuning LLMs using extracted representations, the formulation of a novel ARE framework that efficiently edits LLMs by leveraging representation engineering techniques, and extensive experiments demonstrating the effectiveness of ARE in various editing and censoring tasks. \n\nThe findings show that ARE significantly enhances the safety alignment of LLMs, reduces harmful prompts, and achieves state-of-the-art accuracy in TruthfulQA.", "questions": "Please refer to the weaknesses mentioned above for my questions.", "limitations": "I cannot explicitly find the limitations the authors discussed in their paper. It would be better to merge them into a specific section in the appendix for clarity.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The Adversarial Representation Engineering (ARE) framework offers a practical and interpretable approach for editing LLMs. The iterative process between the generator and discriminator effectively enhances specific concepts within LLMs.\n- The paper addresses the urgent problem of understanding and controlling LLMs' internal mechanisms. It provides a promising solution for safety alignment and hallucination reduction, highlighting limitations in existing methods.\n- The paper is well-written, with clear explanations and concise illustrations. It effectively communicates the motivation, problem formulation, and the proposed ARE framework, making it accessible to readers.\n- Experiments demonstrate the practicality of ARE for editing and censoring tasks. The results showcase enhanced safety alignment, state-of-the-art accuracy, and valuable insights. Code is also released.", "weaknesses": "- The reliability of the concept discriminator should be evaluated. I think conducting some human annotations would be beneficial to check its performance.\n- Should the proposed concept discriminator also be compared against another baseline, where the discriminator simply accepts a text and categorizes whether it can be contained within a concept?\n- I'm not an expert in attack and defense, but I have noticed that recent works on multi-step jailbreaking have gained popularity. Should this also be compared as a baseline?\n  - Li, H., Guo, D., Fan, W., Xu, M., Huang, J., Meng, F., & Song, Y. (2023, December). Multi-step Jailbreaking Privacy Attacks on ChatGPT. In Findings of the Association for Computational Linguistics: EMNLP 2023 (pp. 4138-4153).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer Ga6W,\n\nWe truly appreciate your valuable and constructive comments. We prepared a detailed response to address your concerns in the following.\n\n---\n\n**W1**: The reliability of the concept discriminator should be evaluated. I think conducting some human annotations would be beneficial to check its performance.\n\n**A1**:  We would like to clarify that our goal is not to learn a reliable concept discriminator. Rather, our method aims to learn some discriminator along the way for the goal of effective model editing. Therefore, the effectiveness of the discriminators that we learned along the way is partially evident through the effectiveness of our editing method. Similar to the idea in GAN [1], the discriminator is not on the token or data level but trained on the model output and target. In this framework, the generator is trained to mislead the discriminator to let it not divide generated output and target output (in this paper, the generated representation and target representation), and the target of the discriminator is to discriminate **model output** and **target output**. Ideally, the discriminator should have *no ability* to discriminate the generated representation from the target representation. To this end, we can find out that it is not feasible to have some human annotations and there is actually no need to have a reliable discriminator because 1) the adversarial training process is highly interactively automatic and correlated, thus there is no need for human intervention and 2) we don’t always need a reliable discriminator in this framework as it should finally be misled when the generated representation is quite close to the target ones.\n\n[1] Generative Adversarial Networks, Ian J. Goodfellow et al.\n\n---\n\n**W2**: Should the proposed concept discriminator also be compared against another baseline, where the discriminator simply accepts a text and categorizes whether it can be contained within a concept?\n\n**A2**: Again, we would like to clarify that our goal is not to learn a reliable concept discriminator. Rather, our method aims to learn some discriminator along the way for the goal of effective model editing. Our work profits from the idea of the generative adversarial network (GAN), and the original idea needs gradient for the composed model $D \\circ G$ correlated to the data point. The idea you proposed is straightforward, although there are researches that provides evidence that this does not work for editing LM (see [1]), and on the other hand, if the discriminator is on the token level, then it is not clear how to transport the gradient through the token, so it is hard to control the variables for fair comparing.\n\n[1] Language GANs falling short, ICLR 2020\n\n---\n\n**W3:** I'm not an expert in attack and defense, but I have noticed that recent works on multi-step jailbreaking have gained popularity. Should this also be compared as a baseline?\n\n- Li, H., Guo, D., Fan, W., Xu, M., Huang, J., Meng, F., & Song, Y. (2023, December). Multi-step Jailbreaking Privacy Attacks on ChatGPT. In Findings of the Association for Computational Linguistics: EMNLP 2023 (pp. 4138-4153).\n\n**A3**: Thanks for pointing this out. The provided paper focuses more on privacy information extraction, but our attack is primarily based on alignment-related issues, such as making aligned models provide malicious and misleading information. Therefore, these are not quite the same attacks and are hard to compare.\n\nFor multi-step alignment attacks, the included baselines are already multi-step, like GCG/AutoDAN, which both take multiple steps to optimize the prefix. For multi-shot attacks, we also included ICA as a baseline, which can be designed with multiple prompts and multi-step processes.\n\n---\n\n**Limitations**: I cannot explicitly find the limitations the authors discussed in their paper. It would be better to merge them into a specific section in the appendix for clarity.\n\n**A4**: Thanks for pointing this out. We will merge all limitations, along with the possible social impact and corresponding measurements, into a single section in the Appendix after publication. Our method does have some limitations, such as unstable performance and reliance on human/AI-crafted sequence datasets for the target concept. We will provide a comprehensive discussion then.\n\n---\n\nWe truly appreciate your valuable and detailed feedback. If you have any further questions or concerns, please let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Following a fruitful discussion, the reviewers generally agree that this paper is significant, in that it tackles an important problem; that the proposed framework (ARE) is interesting and creative; and that the empirical evaluation (including the results carried out for the rebuttal) provides convincing evidence of its usefulness.

I encourage the authors to integrate all reviewers’ suggestions, and specifically a thorough limitations section, in the manuscript.
Decision: Accept (poster)