Adversarial Representation Engineering: A General Model Editing Framework for Large Language Models

Authors: yihao  zhang
zeming  wei
jun  sun
meng  sun
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 22:46:13

Abstract: Since the rapid development of Large Language Models (LLMs) has achieved remarkable success, understanding and rectifying their internal complex mechanisms has become an urgent issue. Recent research has attempted to interpret their behaviors through the lens of inner representation. However, developing practical and efficient methods for applying these representations for general and flexible model editing remains challenging. In this work, we explore how to leverage insights from representation engineering to guide the editing of LLMs by deploying a representation sensor as an editing oracle. We first identify the importance of a robust and reliable sensor during editing, then propose an \textbf{A}dversarial \textbf{R}epresentation \textbf{E}ngineering (\textbf{ARE}) framework to provide a unified and interpretable approach for conceptual model editing without compromising baseline performance. Experiments on multiple tasks demonstrate the effectiveness of ARE in various model editing scenarios. Our code and data are available at \url{https://github.com/Zhang-Yihao/Adversarial-Representation-Engineering}.

Main paper: 
 introduction: 
 While Large Language Models (LLMs) have achieved remarkable success in a variety of tasks [34] , their complex internal mechanism makes interpreting and censoring their behaviors (e.g., for safety alignment or hallucination reduction) challenging. To improve the interpretability and consequently the safety of LLMs, numerous efforts have been dedicated to interpreting the internal mechanisms from various perspectives like feature attribution [8, 40] , neuron analysis [12, 33] , and self-explanation [17] .Recently, Zou et al. proposed the idea of Representation Engineering (RepE) [69] , which offers a way of understanding how LLMs work internally by focusing on the overall feature representations rather than individual neurons. Specifically, RepE extracts and analyzes the intermediate features of different concepts (e.g., honesty, fairness, and harmlessness), enabling the monitoring of the internal behaviors of LLMs. More relevantly, RepE potentially allows editing and controlling the behaviors of LLMs by directly intervening in the internal hidden layers during inference. However, as RepE was essentially proposed to monitor the behaviors of LLMs, their proposed method for editing the model through representation vector incorporation is rather limited for practical uses. For instance, their method could disrupt the underlying structure of general LLMs, potentially hindering the model's performance. Additionally, the representation vector used for model editing may not be robust and heavily reliant on carefully chosen hyper-parameters, due to problems such as overfitting.To address these shortcomings, in this work we investigate ways to efficiently fine-tune the model using the representations provided by RepE to achieve specific editing goals. Specifically, we attempt to train an oracle discriminator with the extracted representations given a particular goal of editing Figure 1 : An illustration of our proposed ARE framework. This example showcases how ARE can enhance the concept of "angry" within an LLM. The process involves an iterative dance between the generator and the discriminator. The generator produces outputs, while the discriminator refines its internal representation of "angry" based on these outputs. Through this back-and-forth training, the LLM gradually learns to produce outputs that align better with the concept of "angry." (e.g., harmlessness and trustfulness), then investigate how to use the discriminator to efficiently learn reliable representations and subsequently edit the model accordingly. However, we found that the trained discriminator may (expectedly) fit non-robust features [18] and not be reliable for fine-tuning the models. Therefore, inspired by adversarial learning paradigms like GANs [10] , we extend our idea to conduct adversarial training between the generative model and the discriminator to improve the reliability of the oracle model.Motivated by these studies, we propose an Adversarial Representation Engineering (ARE) framework, utilizing the internal representations and adversarial learning from the generative model and the discriminator, as illustrated in Figure 1 . ARE efficiently and effectively edits LLMs by leveraging representation engineering techniques. In each epoch, it performs two key steps. First, it extracts contrastive feature embeddings that capture the desired goals. Secondly, it simultaneously trains both the LLM and the discriminator model. More details are discussed in the subsequent sections.We conduct extensive experiments to evaluate the effectiveness of ARE on various editing and censoring tasks, including editing the alignment and honesty abilities. Specifically, on one hand, ARE can be used to enhance the safety alignment of existing LLMs effectively; on the other hand, it could also be used to easily remove the alignment for red-teaming goals as well. Compared with some existing fine-tuning-based methods [37, 56] , ARE can substantially decrease the refusal rate on harmful prompts from 20% to less than 1% on Llama2 [46] . Additionally, our ARE fine-tuned model can achieve the state-of-the-art TruthfulQA [26] accuracy. These results present strong evidence of the practicalities of ARE in terms of editing and censoring LLMs.Our contribution in this work can be summarized as follows:1. We propose the Adversarial Representation Engineering (ARE) framework, a novel and efficient method for model editing in LLMs. This framework enables flexible and bidirectional editing, facilitating both the enhancement and removal of specific concepts without compromising the baseline performance of the model.2. We evaluated the ARE framework by addressing critical trustworthiness concerns such as alignment in jailbreak scenarios and hallucination control. Our framework successfully demonstrated its ability to both enhance model alignment and reduce hallucinations, providing robust solutions for improving LLM safety.3. Our method sheds light on the internal workings of LLMs during the editing process, offering a level of transparency compared to traditional fine-tuning, which operates as a black box. By encoding concepts with contrasting sequence pairs and monitoring a small discriminator model, we gain insight into how specific layers are edited to maximize target representations, offering a more explainable approach to model tuning.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: The abstract and introduction accurately reflect the paper's contributions and scope, including the development and effectiveness of the Adversarial Representation Engineering (ARE) framework.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Representation Engineering. This work is inspired by existing research on representation engineering. Since the significant capability of LLMs has sparked great research interest in understanding their internal mechanisms [64, 41, 47] , Representation engineering (RepE) [69] , which seeks understanding and controlling representations of high-level cognition in LLMs, has revealed that there exist low-rank representations that can steer and control specific model capacity. Similar observations are also made in some specific scenarios, e.g. harmfulness [48, 66] and trustfulness [2] . However, RepE did not provide a general solution to edit the model in a practical manner. Recently, the ReFT [53] (Representation Fine-Tuning) method has been proposed as a technique for fine-tuning models based on representation engineering. While ReFT has shown promise as a fine-tuning technique in the realm of representation engineering, it is not specifically designed to excel in model editing tasks.Adversarial Learning. Our proposed method adopts adversarial learning intuitions to improve the reliability of representation discriminators. Adversarial training methods [30, 58, 49, 60, 32] , which optimizes the min-max optimization objective with worst-case performance, was first designed for improving the robustness of machine learning models against adversarial examples [44, 10, 5] . In addition to the adversarial scenario, adversarial training has the benefit of making the representation and prediction more reliable [11, 1] and interpretable [38, 42] , thus also been leveraged in other learning paradigms like image generation (GAN) [10] , domain generalization [9, 43] and contrastive learning [20, 65] for more robust representation. Our proposed framework also leverages the adversarial learning paradigm to make the oracle representation discriminator more robust and reliable.Parameter-Efficient Fine-tuning. This work is related to parameter-efficient fine-tuning. Given the extremely large number of parameters in LLMs, parameter-efficient fine-tuning methods (PEFTs) are designed for tuning the LLM to be adapted to specific tasks with admissible computational overheads. Existing PEFTs can be mainly categorized as (1) module-based, which trains an extra small module in the model, like low-rank adaption (LoRA) [14, 28] and Adapters [13, 35] , and (2) prompt-based, which optimizes a prompt or embedding for the task [39, 23] . While most PEFTs are designed for specific tasks, how to efficiently edit the model knowledge and style remains underexplored [31, 57] .Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We have discussed main assumptions and limitations theoretically in the Notations and Problem Formulation part, and practically in the Experiment part. We have also provided a single limitation section in Appendix.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
notations and problem formulation: 
 In this work, we focus primarily on LLMs, specifically decoder-only architectures, denoted as M (θ) where θ represents model parameters. The model M is structured into several layers, collectively represented by the set L ⊆ N, where each element l ∈ L corresponds to the l-th layer of the decoder.Representations. During the model M processes an input (prompt) x to generate outputs, it also provides representations in the hidden layers. These hidden states can be formulated as H x (•), where H x (l) ∈ R n specifically refers to the representation from the l-th hidden layer when the model processes input x. This architecture forms the basis for our analysis and further discussions in our work. Moreover, the response generated by the decoder model can be denoted as M θ (•) : S → S, where S denotes the set of all valid sentences.Concepts. Next, we define a concept c as the editing goal in the following. A concept applies to the responses generated by the model M . Specifically, we introduce a judge function J c (•) : S → {0, 1} to determine whether an input s ∈ S aligns with the concept c (ideally judged by human oracle). For example, for the concept "angry", J c (x) outputs 1 if a response x is expressed in an angry manner, and 0 otherwise. In addition, for every concept c + , there is a corresponding negation c -which the judgment function is defined as the negation of that for c + , i.e. J c-(s) = 1 -J c+ (s) for all s ∈ S.We are now ready to define the task of conceptual model editing.Assuming that the input prompts follow some implicit distribution D defined on the space S, the task of conceptual editing, aimed at enhancing the concept c + , is to fine-tune the model such that the response r satisfies J c+ (r) = 1 for most inputs. This task is formally defined as arg maxEQUATIONIn general, it is infeasible to edit these concepts directly due to the inability to access the true distribution D or to implement the judgment function J c perfectly. Therefore, a practical approach is to use a curated set of prompts to approximate these abstract concepts. This set of prompts is referred to as anti-target inputs, denoted I A . Accordingly, our training objective becomes arg maxEQUATIONTo effectively demonstrate the target concept c + , we gather a set of prompts known as target inputs I T , which ideally trigger responses consistently exhibiting the target concept, such that ∀x ∈ I T , J c+ (x) = 1. While exhibiting the target concept perfectly may not be feasible, the performance is expected to fulfill the following condition:EQUATIONFor example, consider the target concept of "anger" that we wish to attain (as illustrated in Figure 1 ). To construct the anti-target inputs, we would gather a set of neutral prompts. Subsequently, to obtain the target inputs, we append the suffix "respond in an angry manner." to each prompt. This modification aims to reliably trigger responses that exhibit "anger", thereby constituting an effective set of target inputs.Representation extraction from concepts. Since we have utilized the target input set I T to illustrate the target concepts, the practical objective of fine-tuning shifts towards aligning the responses generated from I A as closely as possible with those from I T . However, achieving token-level similarity is complex and overly fine-grained. Therefore, we employ a high-level approach known as representation engineering (RepE) [69] , which involves manipulating the representations, i.e. outcomes of an embedding function that maps the internal neural activities of each layer into the representation space R n . For any given concept c, the concept can be separated as a distinct feature set apart within this representation space of R n , as examplified in Figure 3a . The process of extracting these representations involves selecting tensors from the hidden states produced by processing an input x across specified layers L e ⊂ L. This process can be formally described by the mapping function R : [H x (l)] l∈Le → R n , which transforms input space S to representation space as a subset of R n . A practical method for implementing this is to concatenate the hidden states from some selected layers.By using these high-level representations, specifically target representations R T = {R(x)|x ∈ I T } and anti-target representations R A = {R(x)|x ∈ I A }, we redefine our optimization goal. Representation serves as a proxy for the concept's embedded features, enabling the definition of a similarity function L M (θ) (•, •) that quantifies the differences between these two sets of representations. The training objective is therefore established as arg minEQUATIONIn the next section, we delve deeper into the methods employed to achieve this objective. In particular, we show that the loss function L effectively functions as a discriminator. 
proposed method: 
 As discussed, the approach suggested in RepE [69] that focuses on generating a target representation vector may be unreliable and overfitted. To bridge this gap, we propose to train a representation discriminator to learn robust representations in an adversarial learning manner. This discriminator, embodied by a neural network, implicitly signifies the representation through the target concept. By iteratively updating this discriminator and the original model, we can facilitate a more refined and robust representation discriminator, forming the core of Adversarial Representation Engineering (ARE) as detailed in the following. Inspired by adversarial learning paradigms like Generative Adversarial Networks (GANs) [10] , ARE employs a dual-model design. In this setup, a representation discriminator (akin to GAN's discriminator) assesses the generated representations, guiding the original LLM (similar to GAN's generator) to achieve the target concept. We show this duality in Figure 2 .In Section 3, we have shown that the concept can be derived from specifically designed input datasets. Note that the goal of editing is to minimize the gap between the representations from the two datasets asR T = {R(M, x)|x ∈ X T } and R A = {R(M, x)|x ∈ X A }.Expressing the difference in features between the two sets above concisely through a single tensor or numerical metrics can be challenging. Therefore, we propose to encode this feature into a classifier in the form of simple neural network models. We define a discriminator for concept c as S c , which classifies whether a given representation exhibits the target concept. It accepts a representation vector and returns the confidence that it exhibits the concept. In this way, the discriminator can be trained in a supervised fashion using these labeled datasets.However, a discriminator trained on such (limited) datasets may not accurately capture the desired representation's feature due to the presence of numerous samples near the decision boundary and adversarial samples. For generalized conceptual editing, we aim to obtain (through the decoder model) a generalized and robust target presentation that works for all inputs. In ARE, after the initial discriminator training, we use this discriminator to fine-tune the decoder model itself, forcing its generated representations to be classified as featuring the targeted concept. Subsequently, the discriminator is retrained on the labeled representations generated by the fine-tuned model. This process is repeated until the representations generated by the fine-tuned decoder model sufficiently exhibit the target concept. The core idea is to allow the decoder model and the discriminator to be adversarial to each other, similar to the approach employed in GAN.The overall editing algorithm is presented in Algorithm 1. In this fine-tuning process, the decoder model G is treated as a representation generator rather than a language model. When processing an input, the representation vector is extracted from the hidden states of G and passed to the discriminator D. Leveraging the Low-Rank Adaptation (LoRA) [14] technique, we edit some selected layers of the generator G to maximize the probability of generating representations classified as the target class by the discriminator D, while keeping the parameters of D frozen. Notably, the gradient can propagate through the entire framework by combining the generator and discriminator into a single model.Input: A representation generator decoder model G that gets a string and returns corresponding representation, representation discriminator D(δ) with parameter δ, targeted inputsEQUATIONT }, anti-target inputsEQUATIONA }, layers to edit L e , layers to gather representations from L r , optimizing epochs T , target concept label y Target , learning rate for discriminator l D Output: Fine-tuned LLM G T class CombinedModel:Generator G; Discriminator D; Forward Propagation Method M (•); M ← CombinedModel(G, D(δ Init )); ▷ M (•) is defined as M (x) = D(G(x)) M lora (θ) ← LoadLoRAAdapter(M, L e ); ▷ Only to edit LoRA parameters θ in layers l ∈ L e for t : 1 → T do R T , R A ← []; ▷ Initialize Representation Dataset for i : 1 → n do R T .append(M lora .Generator(I (i) T )); R A .append(M lora .Generator(I (i) A )); end R ← R T ∪ R A ; update δ by minimizing L(δ) = ∇ δ 1 |R| r∈R -(log 1 r∈R T (r)P D(δ) [y T |r] + log 1 r∈R A (r)P D(δ) [y A |r]); M.Discriminator ← D(δ); ▷ Train the discriminator on labeled R T ∪ R A I ← I T ∪ I A ; update θ by minimizing L(θ) = ∇ θ 1 |R| x∈I -log P M lora (θ) [y Target |x];M lora ← M lora (θ); ▷ Fine-tune G by LoRA to ensure it generates targeted representation end return M lora .Generator;To provide a clear understanding of the alternative training process, we offer a visualization in Figure 3 . We compiled a set of 256 prompts, evenly divided between normal and malicious, with the expectation that the aligned model will reject all malicious inputs. The representations derived from these prompts are plotted using t-SNE, as depicted in the figure. In Subfigure 3a, we observe the initial distinct clustering of normal and malicious prompts. Our goal for model editing is to adjust these representations so that the two types of prompts yield similar responses. During the first epoch, illustrated in Subfigure 3b, the malicious prompts begin to converge towards the cluster of normal prompts. Since the two categories of representations remain distinct, necessitating further refinement of the discriminator. After 30 epochs of iterative training as shown in Subfigure 3c, we observe that the representation of normal prompts remains consistent, having been continuously classified correctly. Meanwhile, the representations of malicious prompts have nearly merged into the normal cluster, making it challenging for the classifier to distinguish them. At this stage, the differences in representations are minimal and can be considered negligible, indicating a successful editing process.In the following, we present details of the editing algorithm in ARE. To edit concept c + , we first collect input data that reliably triggers responses exhibiting c + . Similarly, to train a discriminator for the opposite concept c -, we collect corresponding triggering input data. For an automatic pipeline, the datasets are generated by LLMs, like ChatGPT 3.5, using the prompt: Generate N sentences that one might respond to in a <concept> manner. Approximately 10 2 input prompts per dataset track suffice. During training, we minimize the overall cross-entropy loss of D(G(p)), where p is an input from any category. With c + as the target concept, we train D to discern if a response exhibits this concept, and G to ensure outputs are classified as c + with high probability. This entails a two-step optimization:Step 1. Train D(δ) to fit G by optimizing L(δ): Consider generated target representations R T corresponding to c + and anti-target representations corresponding to c -. The loss L(δ) is defined as the classic cross-entropy loss, which isEQUATIONStep 2. Train G(θ) to fit D by optimizing L(θ): Consider all input prompts in set I. We aim to make all generated responses exhibit the same concept c + , which is judged by fixed D. Thus the loss L(δ) is defined as the cross-entropy loss for the probability of classifying a prompt to c + , which isEQUATION)Gradient descent is applied to optimize the two components as they compete. Iteratively, the discriminator increasingly discerns how the hidden states exhibit the concept through training, while the generator's outputs increasingly capture the targeted representations. Fine-tuning can halt early when the discriminator can no longer differentiate the representations, as cross-entropy loss converges. 
experiments: 
 To evaluate the effectiveness and flexibility of ARE, we apply it to two distinct conceptual editing tasks: jailbreak and its defense, and control of hallucinogenic text generation. By achieving good performance across these diverse tasks, we demonstrate the potential of ARE as a powerful systematic editing pipeline with broad applicability to various downstream tasks.Background. With the application of various safety training techniques [25, 3] , LLMs can often generate responses aligned with human values, but recent research has also revealed the vulnerability of LLMs to adversarial attacks, particularly referred as jailbreaking [70, 50, 61, 7, 19] . These attacks successfully craft malicious prompts that induce them to generate harmful outputs. Recognizing the need for combating such attacks (i.e., blue team) and for evaluating the risk brought by model editing techniques (i.e., red team), we evaluate the potential of applying ARE for editing the concept of alignment, i.e., to either enhance (defend) or remove (attack) the alignment ability of LLMs.Experiment Setup. We evaluate our methods using three open-source, aligned LLMs: Llama-2-7B-Chat [46] , Vicuna-7B [67] , and Guanaco-7B [6] , for both attack and defense tasks. Our discriminator is a 2-layer neural network with a hidden layer consisting of 512 neurons. More details on the training of the discriminator can be found in Appendix A.1. Following [15, 29] , we employ the Advbench dataset proposed by Zou et al. [71] to test our results. This dataset contains about 500 malicious prompts that violate the alignment principles of LLMs. To measure the effectiveness of ARE, we consider three distinct categories of attack techniques as baselines, including 1) templatebased attacks ( In-Context Attack (1 shot) [50] and DeepInception [24] ), 2) optimization-based attacks ( GCG [71] and AutoDAN [29] ), and 3) editing-based attacks (Contrast Vector from RepE, Shadow Alignment [56] and harmful examples demonstration attack (HEDA) [37] ). Note that the optimization-based methods may demand 10 2 to 10 3 times more time to execute compared to others.For the aspect of model defense, Self-Reminder [52] and In-Context Defense [50] are adopted as baseline defense strategies.Experimental Results. Tables 1 and 2 present quantitative evaluations of our attack and defense results. The analysis of attack outcomes reveals that existing jailbreak attacks are not sufficiently effective, as indicated by low attack success rates, rendering them undesired for reliable red-teaming tasks. Conversely, our method, which employs editing-based attacks, demonstrates superior performance over all other listed editing-based approaches, achieving near-perfect success rates (close to 100%) against the most secure models like Llama-2. Some evaluation results of our attack method on larger language models are shown in Table 6 in Apeendix A.3. Furthermore, our analysis of various defense mechanisms against optimization-based attack strategies demonstrates that our modified model substantially improves the alignment and safety of the targeted models. Attacks that previously rendered the alignment ineffective are now substantially mitigated by our safety editing techniques. For instance, the attack success rates have markedly decreased to 41.1% for AutoDAN and 28.8% for GCG attacks on Vicuna. This result suggests that model editing may be considered an effective alternative or complementary approach to standard alignment methods such as RLHF.Background. Extensive prior research has demonstrated that LLMs may produce inaccurate or entirely fabricated information [63, 16, 4] , a phenomenon often termed hallucinations. In this part, we explore the application of ARE for editing the hallucination extents of LLMs. Our objective is twofold: 1) amend LLMs to enhance their accuracy and truthfulness, thereby mitigating hallucinations; and 2) similar to the jailbreak scenario, for red-teaming purposes, to intentionally induce hallucinations. We remark that the latter is useful since there exists a demand to generate hallucinatory responses as a form of deterrent, aiming to discourage the occurrence of hallucinations in future instances [62] .Experiment Setup. In line with experimental protocols established by prior research [69] , we curated target and anti-target prompts from the Alpaca Instruction Tuning Dataset [45] , a dataset contains various instructions that cover multiple fields, hence mitigating task-specific biases. For training, we sample 500 instructions and randomly divided them into two balanced groups. We choose one group of prompts as the anti-target dataset without any perturbation. For the hallucination-enhancing task, the prompts are attached with the prefix "Please give an untruthful answer", inducing the model to produce hallucinated responses; by contrast, for the hallucination-reducing goal, the target dataset was prompted with a prefix "Please give a truthful answer", guiding the model towards accurate and reliable outputs. Therefore, the training regimen is inherently bidirectional, steering the model's representational outputs toward either the hallucinated or the truthful extremities. To demonstrate the versatility of our method without the need for task-specific hyper-parameters and settings, we employed the same settings as delineated in the Jailbreak tasks described in Section 5.1, with the sole variable being the dataset employed.Evaluation Metric. Building upon previous studies [62, 22] , we utilized the TrustfulQA benchmark [26] for evaluating the tendency of models to produce hallucinations, which comprises 817 questions across 38 subcategories, each designed to potentially lead models towards incorrect beliefs, misconceptions, or biased responses. In its multiple-choice format, TrustfulQA provides an average of around 5 options per question, among which only one answer is factual, while the others include hallucinated content. For hallucination evaluation, we adopt Correct Answer Rate (% Right Answer), defined as # Right Answer/ # Answer (more details in Appendix A.2).Experiment Results. We implemented bidirectional editing and benchmarked our approach against recent strategies aimed at mitigating hallucinations, including Self Reminder [52] (prompting the inputs with prefix Please give a/an truthful/untruthful answer) and Inference-Time Intervention (ITI) [22] . The outcomes of these comparisons are detailed in Table 3 . The efficacy of our model is evident, as our hallucination-enhancing editing led to a minimal number of correct responses; conversely, the hallucination-reduction editing significantly surpassed other evaluated approaches in both metrics, demonstrating that ARE effectively addresses hallucinations without diminishing the model's ability to provide valid responses. It is noteworthy that the model, after undergoing the editing process, exhibits improved performance relative to the target input set, showing the efficacy of our method. This enhancement also enables the post-editing model to achieve superior performance on tasks that were previously unattainable.Background. While the two aforementioned sections focus on evaluating how successful the editing is in terms of achieving the target concept, it is essential to assess the naturalness and usefulness of the generated texts. Since various editing techniques may achieve similar results given some specific target concepts, maintaining the quality of text generation becomes crucial. Current editing approaches, which focus on predefined and single target edits, typically produce models plagued by problems such as repetition and notably reduced diversity in the generated texts, as exemplified in Table 4 . For example, the outputs may lack variability, frequently recycling similar structures; for individual responses, sentences might be very similar or entirely repetitive.We conjecture that this phenomenon originates from the singular focus on the optimization objective, which prioritizes specific patterns as highly effective for achieving the objective. In contrast, our method employs a dynamic generation of optimization targets via the alternating training of a discriminator, ensuring that our optimization objectives are both natural and widely relevant.Evaluation Metrics. We leverage several quantitative metrics to assess the quality and diversity of the generated texts, benchmarked by the held-out test set in [56] . Drawing inspiration from prior quality and diversity evaluation of text generation [51, 27, 54] , we adopt Repetition-4 (defined as 1.0 -|unique 4-grams|/|4-grams|) to gauge phrase-level repetition and Repetition-Sen (defined as 1.0 -|unique sentences|/|sentences|) for measuring sentence-level repetition. Additionally, we utilize the Self-BLEU score [68] , which measures the degree of similarity between segments of a text by comparing overlapping phrases within the text itself, serving as an indicator of both the uniqueness and variation in the generated content. Still, the generation process for all responses was standardized as the same as the default parameters used above.Experimental Results. The results of this analysis are tabulated in Table 5 , which compares the efficacy of our editing method against a variety of alternative approaches. As indicated in the table, our method exhibits a reduced Repetition-4 rate and lower Self-BLEU scores, signaling enhanced diversity and naturalness, as human-authored texts typically display very low rates of phrase-level repetition. Following the causal study in RepE [69] , we extract runtime representations at the last token on the hidden states in the last 5 layers of the models. These models are fine-tuned on a dataset provided in RepE [36] , which contains a set of prompts that are labeled either normal or malicious. For fine-tuning, we employ the Parameter-Efficient Fine-Tuning (PEFT) framework [55] , using the Adam optimizer [21] with a learning rate of 1 × 10 -4 . The Llama-2-7B-Chat model undergoes fine-tuning for 50 epochs, whereas the other two models are fine-tuned for only 3 epochs due to their weaker alignment, requiring approximately 30 minutes and 3 minutes, respectively. Specifically, each epoch of fine-tuning takes 1-2 minutes with fewer than 1000 examples, which are usually sufficient for training. For VRAM, it costs the same as the LoRA fine-tuning method, which is fast and needs smaller space as it is a parameter-efficient fine-tuning method.To exploit the advanced capabilities of LLMs like LLaMA-2, which excel in generating responses based on instructions, we diverged from conventional methodologies designed for generative models, which typically rely on calculating the log probabilities for each answer, a process that may lack precision and stray from practical applications. Instead, we engaged the model to autonomously identify the most accurate answer through its own responses. This approach evaluates the model's proficiency in distinguishing between factual content and hallucinations, mirroring real-world situations where individuals derive information from responses rather than underlying probabilistic data. This metric has gained traction in contemporary benchmarks, reflecting its relevance and applicability in assessing model performance in scenarios akin to human information processing [59] . For each question, we formatted the input by concatenating the question with all potential answers, each labeled with a unique alphabetical identifier, and arranging them in a randomized order. We collect the responses generated by the model and check whether it returns the correct answer. A model's propensity of hallucination is measured using Correct Answer Rate (% Right Answer), defined as # Right Answer/ # Answer, which assesses a model's capability to identify the truthful response. We identify two major limitations of our proposed ARE framework.Potential for Misuse. Our framework can be used to bypass safety mechanisms in large language models (LLMs), potentially enabling the generation of harmful or malicious content such as hate speech or misinformation. This presents risks for misuse, and while the method can be used defensively as the editing is bidirectional, further exploration of preventive measures to mitigate negative societal impacts is required.Dependency on Human/AI-Crafted Datasets. Our framework relies heavily on specific sequence datasets that represent target concepts, which may limit its generalizability and performance across different tasks. Developing more adaptive approaches that require less fine-tuned data is an area for future work.NeurIPS Paper ChecklistQuestion: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We provided our code for major experiments in Supplementary Materials.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The paper specifies all the training and test details, such as data splits, hyperparameters, optimizer type, etc., necessary to understand the results, which are presented in the Experiments and Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We have claimed that our experimental data is calculated with multiple repeating experiments and the variant is small, as claimed in Experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics, as the authors have considered ethical implications throughout their work.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: The paper discusses both potential positive and negative societal impacts of the work in Experiments and Limitations in Appendix.• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: We have shown ways for safeguarding pretrained language models from being jailbreaked in our methods.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators or original owners of assets used in the paper are properly credited, and the license and terms of use are explicitly mentioned and respected, as referenced in the Related Work or Experiments.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators.  
 
 Appendix:  


Decision: Following a fruitful discussion, the reviewers generally agree that this paper is significant, in that it tackles an important problem; that the proposed framework (ARE) is interesting and creative; and that the empirical evaluation (including the results carried out for the rebuttal) provides convincing evidence of its usefulness.

I encourage the authors to integrate all reviewers’ suggestions, and specifically a thorough limitations section, in the manuscript.

Reviews: review: paper_summary: paper_summary: This paper explores how to use representation engineering methods to guide the editing of LLMs by deploying a representation sensor as an oracle. The authors first identify the importance of a robust and reliable sensor during editing, then propose an Adversarial Representation Engineering (ARE)  framework to provide a unified and interpretable approach for conceptual model  editing without compromising baseline performance. Experiments on multiple  model editing paradigms demonstrate the effectiveness of ARE in various settings.
strength_weakness: strength_weakness: The technical novelty is somewhat incremental, as the proposed approach can be regarded as applying adversarial training to representation engineering. 

There is no analysis of training efficiency and computational resources. 

Some symbols are used without definition, and the experimental setup is somewhat vague.

There are some missing references:

ReFT: Representation finetuning for language models

Editing Large Language Models: Problems, Methods, and Opportunities
questions: questions: See weakneses.
limitations: limitations: No
paper_summary: paper_summary: This paper addresses the challenge of understanding and controlling the internal mechanisms of Large Language Models. It proposes a novel Adversarial Representation Engineering (ARE) framework that leverages representation engineering and adversarial learning techniques. The proposed framework aims to provide a unified and interpretable approach for conceptual model editing without compromising baseline performance. The key contributions of this paper include the introduction of a representation engineering framework via adversarial learning like GAN, and the conducted experiments has demonstrated the effectiveness of ARE in a series of editing and censoring tasks.
strength_weakness: strength_weakness: 1. The paper does not provide extensive discussion on the scalability of the proposed method for extremely large models (larger than 7B), which could be a practical limitation.
2. In the experiment section for Hallucination, only one baseline (self-reminder) is compared, which may be not enough to show its advantages over other methods like TruthX, etc.
questions: questions: It is stated that ARE is a unified and interpretable approach for conceptual model editing, however, I can't see the interpretability of ARE after reading the paper. Can you provide more explanation?
limitations: limitations: As the proposed ARE framework can be used to edit models to bypass safety mechanisms and generate harmful or malicious content, which may be utilized to produce misleading information or hate speech, the potential measurements to prevent its negative social impact could be discussed.
paper_summary: paper_summary: The paper proposes an Adversarial Representation Engineering (ARE) framework to address the challenge of editing Large Language Models (LLMs) while maintaining their performance. 

The authors introduce the concept of Representation Engineering (RepE) and extend it by incorporating adversarial learning. 

The main contributions include the development of an oracle discriminator for fine-tuning LLMs using extracted representations, the formulation of a novel ARE framework that efficiently edits LLMs by leveraging representation engineering techniques, and extensive experiments demonstrating the effectiveness of ARE in various editing and censoring tasks. 

The findings show that ARE significantly enhances the safety alignment of LLMs, reduces harmful prompts, and achieves state-of-the-art accuracy in TruthfulQA.
strength_weakness: strength_weakness: - The reliability of the concept discriminator should be evaluated. I think conducting some human annotations would be beneficial to check its performance.
- Should the proposed concept discriminator also be compared against another baseline, where the discriminator simply accepts a text and categorizes whether it can be contained within a concept?
- I'm not an expert in attack and defense, but I have noticed that recent works on multi-step jailbreaking have gained popularity. Should this also be compared as a baseline?
  - Li, H., Guo, D., Fan, W., Xu, M., Huang, J., Meng, F., & Song, Y. (2023, December). Multi-step Jailbreaking Privacy Attacks on ChatGPT. In Findings of the Association for Computational Linguistics: EMNLP 2023 (pp. 4138-4153).
questions: questions: Please refer to the weaknesses mentioned above for my questions.
limitations: limitations: I cannot explicitly find the limitations the authors discussed in their paper. It would be better to merge them into a specific section in the appendix for clarity.

Comments: comment: Dear Reviewer,

I would appreciate if you could comment on the author's rebuttal, in light of the upcoming deadline.

Thank you,
Your AC
Thanks for your reply and the demonstration of more experiment results, I have increased my score.
Thank you for your reply, my concern has been addressed. I have raised my score.