Inspecting and Editing Knowledge Representations in Language Models
Authors: Evan Hernandez
Belinda Z. Li
Jacob Andreas
Venue: COLM (2024)
Keywords: representation editing, knowledge, factuality, interpretability, world models
TL;DR: Introduces representation editing for LMs and applies it to controlled generation, knowledge editing, and failure detection

Abstract: Neural language models (LMs) represent facts about the world described by text. Sometimes these facts derive from training data (in most LMs, a representation of the word *banana* encodes the fact that bananas are fruits). Sometimes facts derive from input text itself (a representation of the sentence *I poured out the bottle* encodes the fact that the bottle became empty). We describe REMEDI, a method for learning to map statements in natural language to fact encodings in an LM's internal representation system. REMEDI encodings can be used as *knowledge editors*: when added to LM hidden representations, they modify downstream generation to be consistent with new facts. REMEDI encodings may also be used as *probes*: when compared to LM representations, they reveal which properties LMs already attribute to mentioned entities, in some cases making it possible to predict when LMs will generate outputs that conflict with background knowledge or input text. REMEDI thus links work on probing, prompting, and LM editing, and offers steps toward general tools for fine-grained inspection and control of knowledge in LMs.

Published as a conference paper at COLM 2024
Inspecting and Editing Knowledge Representations in Language Models
Evan Hernandez, Belinda Z. Li & Jacob Andreas
Computer Science & Artificial Intelligence Laboratory Massachusetts Institute of Technology (dez,bzl, jda)emi @..tmmii.eee edu
Abstract
Neural language models (LMs) represent facts about the world described by text. Sometimes these facts derive from training data (in most LMs, a representation of the word banana encodes the fact that bananas are fruits). Sometimes facts derive from input text itself (a representation of the sentence poured out the bottle encodes the fact that the bottle became empty). We describe REMEDI, a method for learning to map statements in natural language to fact encodings in LM's internal representation system. REMEDI encodings can be used as knowledge editors: when added to LM hidden representations, they modify downstream generation to be consistent with new facts. REMEDI encodings when compared to LM representations, they reveal which properties LMs already attribute to mentioned entities, in some cases making it possible predict when LMs will generate outputs that conflict with background knowledge or input text. REMEDI thus links work on probing, prompting, and LM editing, and offers steps toward general tools for fine-grained inspection and control of knowledge in LMs.
also be used as probes:
Introduction
PATCHING ERRORS
laweyor
Neural language models (LMs) build implicit, structured models of the state of the world: their representations encode general knowledge (Petroni et al., 2019) and situations described in input text (Li et al., 2021). Sometimes these representations contain mistakes, resulting in er- rors in generated text (Fig. 1). As LMs improve, versions of these problems are likely to persist: large LM training sets contain erroneous and contradictory information, go out of date, and harbor unexpected biases (Bender et al., 2021). Even in domains where LM generation is more reliable, understanding how model-internal rep- resentations relate to output is crucial for attribu- tion and controlled generation (Akyürek et al., 2022; Dai al., 2022). There is thus a fundamen- tal need for techniques that can inspect and edit LMss' knowledge, whether derived from training data or input text.
REMEDDI
Anita's 's faw office ser the lower Easiern Shore inchuling
Accomack County. a-narane
[IM completion)
an ultorney. [REIEE catsletion)
Incuted Arizona
UPDATING FACTS
REMEDI
crass London Bridge, should travel tor the South Bankh. Arizona.
located London: 0.3
DETECTING INCOMERENT GENERATION
located Milan:
0.7
REMEDI
heudquarieu Versuce gurnaunded by London, Versace's 's headquarters surrounded by restaurants including the famous 'Casa Youie the Milan.
Figure 1: REMEDI can patch errors made by LM and insert new facts with or without context provided in the prompt. It can also help detect errors before gen- eration.
This paper introduces REMEDI (REpresentation MEDIation), a technique for discovering directions in LM-internal representation spaces corresponding to encodings of factual attributes (like is a lawyer in Fig. 1). When these encodings are added to LMs' representations of entities (like Anita), they edit the facts that LMs attribute to those entities some cases producing output that cannot be produced with a corresponding textual prompt. Encodings produced by REMEDI can also be used to interpret LMM representations, making it possible

---

Published as a conference paper at COLM 2024
to probe LMs' factual knowledge, and to predict when they will generate incorrect or incoherent output.
Even when trained only to modify LMs' background knowledge, REMEDI generalizes to tasks that require querying and modifying knowledge specified in-context. Our findings thus suggest that LMs represent and integrate information from these sources in unified manner. REMEDI offers steps towards tools that can monitor and control language generation by directly specifying facts and situations in an LM's native encoding scheme."
2 REMEDI
Motivations: control and interpretability Consider the examples from Fig. (top). In the first example, the LM prompted with the text Anita's law office seroes the lower Eastern Shore..., which provides some context about the entity Anita. However, when the LM generates a continuation of this prompt, it asserts that Anita is a nurse. We term this incoherence failure of context integration: information provided in the textual context has failed to alter the LM's predictions. It would be useful to identify and fix such errors, changing model's ' encoding of entities like Anita to ensure that she is correctly described as an attorney. In addition to ensuring discourse coherence, it is often desirable to modify prior associations in LMs. In Fig. (middle) the LM strongly associates London Bridge with the city of London because the most famous London Bridge is located there. However, there could be (and are²) other London Bridges, and we might wish to control an LM to make the lesser-known bridge more salient.
It is sometimes possible to achieve these goals by carefully prompting models with the right input text. But due to the non-systematic, opaque nature of prompt engineering (Jiang et al., 2020b), sigmificant manual effort is often required to find prompt (if one exists at all) that yields correct behavior and generalizes to new use cases. Techniques for localizing generation failures within LMs' internal representations would make it possible detect them in advance, and guide research aimed at mechanistic understanding of the relationship between LMs' internal representations and their textual outputs.
Overview At a high level, our proposed approach learns how to intervene in an LM's representation space to modify the LM's knowledge about a mentioned entity (like Anita in Fig. 2). This intervention ultimately updates the LM's representation of the entity to encode an attribute (e.g., is a lawyer) so that the LM will generate text about the entity consistent with the new attribute. This update operation can be specified by a single vector, and is applied to the hidden representation of single token at single layer. Edits produced by REMEDI can also be applied out-of-context (enabling controlled generation without textual prompts). By comparing edit vectors to unedited representations, REMEDI additionally makes it possible to inspect representations of entities and attributes produced during ordinary model operation.
Editing representations Assume we have language model (xM that assigns probabili- ties to strings x consisting of tokens Xin- In this paper, PLM will always be an autoregressive transformer (Vaswani et al., 2017) pretrained on English text, as in the GPT family of models (Radford et al., 2019; Brown et al., 2020). These models decompose p(x) into a product of next-token distributions given preceding context: PLM (x) II ( X1s-1). Our goal is to modify an LM's internal state to cause it to generate desired text about a target entity.
Where and how should we perform this modification? LMs encode factual information in hidden representations of entity mentions: entities' states (Li et al., 2021), perceptual
1 Code and data are available at rttiiiallaatttttee/.h..eeeeeeeee
2Such as the one in Lake Havasu City, Arizona.
'These issues are not solved with scale: "prompt injection attacks" that cause LMs to ignore initial instructions (Perez & Ribeiro, 2022; Greshake et al., 2023) may also be viewed as failures of context integration, and might (beyond the scope of this paper) also be mitigated with better tools for directly manipulating representations of tasks rather than facts.
2

---

Published as a conference paper at COLM 2024
features (Abdou et al., 2021), and other semantic properties (Grand et al., 2018) have been shown to be linearly decodable from entity representations. To ensure that an LM encodes the fact Anita is lawyer, it should thus suffice to find an appropriate transformation of the representation of the token Anita.
Formally, denote the transformer's hidden representation for token x; in layer as k/ and we write (L In z) to mean the output of PLM with n!! "hard-coded" to equal z during the forward pass. Given representations of the entity Rentity and the target attribute Nattr, REMEDI specifically learns an affine transformation that returns new entity representation according to:
z = F(hhenity, Matte) Hentity + Whattr + b.
(1)
such that when 2 replaces the entity inside the LM, the LM will generate text consistent with the target.
How should we pick haur, W and b? Building on the success of linear probing approaches (Con- neau et al., 2018; Belinkov & Glass, 2019), it is tempting to begin by training classifier for the presence or absence of attributes. For example, following Li al. (2021), we could take hattr to be the LM's own representation of an attribute (like plays the oboe; Fig. 2), then optimize W and b to predict whether an entity representation en- codes the attribute:
Ronnty
pred
LM
encodinng
went work
LM GENERATION
Maitr
Represantation editor (REVEDI)
Edit
encoding
Neavy" Wh
plays the obve
LM pred
the concort hall
Eyg
LM encoding
p(attribute entityy = o o'hentity Whattr b) (2)
John
went
work
REMEDI EDITING
However, even when an LM encodes infor- mation in its representations, this information may not causally influence subsequent genera- tion (Ravfogel et al., 2020; Elazar et al., 2021; Ravichander et al., 2021). An effective editor must identify fact encodings that are causally linked to output.
Figure 2: Illusraaion of REMEDI. Given an prompt (John went to work at) and desired attribute (plays the oboe), REMEDI constructs an edit to the hidden repre- sentation of John that increases the prob- ability of an appropriate completion (the concert hall).
Learning Effective Edits REMEDI optimizes W and b to directly intervene in LM. We assume access to a dataset of tuples (Xim-1, fentiyy fattr, figt), where X1:n-1 is a textual context John went to work at), lentity is the index of an entity within the context, fattr is the attribute to be inserted (plays the oboe), and figt is a generation target: a completion that should be assigned high probability if the attribute is applied to Xentity (the concert hall). Following Li et al. (2021), we obtain representation hattr by averaging the LM's encoding of fattr. We train the editor F to maximize the probability that PLM assigns to figt after modifying the hidden representation of Xentity(Fig. 2):
Lig((z) log (LM = figt Xim- Mentity = =2)
Learning Non-Destructive Edits When LMS encode strong prior associations between entities and properties (e.g,, in the London Bridge example; see Hase et al., 2021), it is necessary to remove these facts while inserting new ones. We obtain a target string fprior assigned high pre-edit probability, and train minimize the probability of prrorr
Lprir (z) = log ( fprior X1.nit, = =z)
(4)
Finally, to prevent the degenerate solution in which the language model always (and only) predicts figt, we penalize the language model for changing its distributions on all tokens between the entity mention and the time at which it predicts fig::
N Σ
(z)
DKI. (PIM(- Xei hentity = z) PIM(- (xci))
'Henefoorth we will omit the layer index, but the reader should always assume all operations occur a single layer.
3

---

Published as a conference paper at COLM 2024
Unlike the distribution over tokens at the end of the prompt, which should change dramat- ically under the intervention, the distribution over these intermediate tokens should not change significantly. LKL penalizes such changes. Thus, the complete objective function is:
((z) Ligt(2) A Arrorr + A2CxL(z),
(6)
where A and A2 are hyper-parameters. We evaluate REMEDI by studying its ability to control model output (Section 4) and to interpret and predict model behavior (Section 5).
Related Work
Probing factual knowledge Large language models (LLMs) trained on massive text datasets have been shown to encode context-agnostic factual knowledge, which can be queried through text prompt (Petroni al., 2019). Most work on extracting background factual knowledge from LMs focuses on designing textual queries for different sources of knowledge (Richardson & Sabharwal, 2020; Peng et al., 2022). Additionally, knowledge probes may sometimes recover factual information even in cases when LMs do not generate truthful outputs with high probability (Burns et al., 2022).
Probing representations of individual situations Neural LMs have also been shown to build representations of context-dependent knowledge. (Li et al., 2021) show that they track aspects of entity state over discourse, and this state can be extracted from LM represen- tations of contextualized entity tokens. Furthermore, many LMs have been (indirectly) evaluated on their ability to track context-dependent knowledge by having their perfor- mance measured on downstream reading comprehension tasks in wich the LM is expected to answer questions about facts within a discourse. Reading comprehension datasets such as CoQA (Reddy et al., 2019), RACE (Lai et al., 2017), and SQuAD (Rajpurkar et al., 2016) are now part of the standard evaluation suite for new LMs; and most modern LMs perform well (Brown et al., 2020). However, generating does not always imply knowing. Datasets con- tain spurious correlations (Gururangan et al., 2018), and LMs are sensitive to the phrasing of prompts and questions (Jiang et al., 2020b).
Editing LLMs In the past, LLMs have been predominantly adapted to new tasks and knowledge through fine-tuning (Devlin et al., 2019). Recently, with very large LMs, new classes of adaptation methods have been introduced, which generally fall into one of the following two categories: (1) Prompt design approaches prepend a textual prompt to each example specifying the adaptation target (Brown al., 2020). (2) Prefix-tuning approaches prepend continuous learned tokens ahead of each example. These specify a task for the LM similarly to how a textual prompt might (Li & Liang, 2021; Lester al., 2021). Control token approaches similarly use these learned tokens to controls aspects of LM output, including sentiment (Dathathri et al., 2020), style (Keskar et al., 2019), and semantics (Ross et al., 2022). Prompts can be fragile; LMs may fail to generate text consistent with the prompt, as in Fig. 1.
Finally, large body of work examines how to localize and edit factual information an LM's parameters (Meng et al., 2022a;b; Mitchell et al., 2022; Dai et al., 2022). For example, ROMÉ (Meng et al., 2022a) localizes factual knowledge in LMs a particular subset MLP modules, and edits specific facts in targeted way through rank-one modification of MLP weights. Unlike REMEDI, these approaches operate on models' weight matrices rather than representations, meaning they can correct errors in models' background knowledge but not information provided context.
4 Controlling Generation
We begin by showing that the REMEDI procedure described in Section is an effective tool for controlling LM output. Intuitively, if REMEDI succeeds in creating a new entity representation encoding the desired attribute, text generated by the LM about the entity should at minimum (a) prefer generations consistent with the target attribute over potentially contradictory attributes and (b) remain as fluent the original generations. Our experiments in this
4

---

Published as a conference paper at COLM 2024
section test properties (a) and (b), as well other quality measures, in two different settings. In the first setting, we use REMEDI to patch incoherence errors, editing the LM to reinforce the information provided in the context. In the second setting, use REMEDI to update prior knowledge about entities (such as the Versace Headquarters example in Fig. 1). A third set of experiments, described in Appendix D, evaluates an additional word re-definition task. These experiments show that REMEDI often successfully controls model behavior even when prompting fails. It can thus serve as a building block for future controlled generation interfaces that allow users to directly steer model behavior in representation space.
4.1 Patching Errors
We first use REMEDI to manipulate representations of generic named individuals, such as Anita Dennis, about whom the LM should have no prior association (and about whom the LM should acquire all information from the prompt). We provide a small amount of context about each person and prompt the LM to predict their occupation from small set of candidates. As we will show, the LM often completely ignores this context, and prefers unrelated occupations to ones highly relevant given the context (nurse attorney in Fig. 1).
Setup In this and all following experiments, we use GPT-J as the underlying language model (Wang & Komatsuzaki, 2021) and include results for two additional models in Appendix E. GPT-J is a 6B parameter, decoder-only transformer pretrained on the Pile (Gao et al., 2020). We obtain biographical sentences from the Bias in Bios Dataset (De-Arteaga et al,, 2019). This dataset consists of =397k short professional biographies people scraped from the internet. Each biography is paired with label for the subject's occupation. We take one sentence from each biography (details in Appendix B), using only the subject's first name, and prompt the LM with the biographical sentence followed by (Person) ) has the occupation of. We then look at the relative probabilities of 28 occupations under the LM, and consider output correct if the true occupation is ranked first. GPT-J succeeds about half the time (55%, In-context baseline in Table 1) on this task. Table 2 shows example errors.
Method We use REMEDT to create new rep- resentations of the first-name-only entities encoding the target occupation. We take Mentity to be the last token of the last entity mention (right before model predicts the occupation), and we take hattr to be the av- erage representation of the biographical sen- tence after the entity. Note this means we are not using any additional data to construct the new entity-the input to REMEDI is all text provided in context to the LM. We train the editor on 5000 examples using Eq. (6), with figt set to the target occupation and with no fprior term (A1 = 0). Edits are per-
In Context Fluency
No Context Fluency
Method LM-only REMEDI
Acc.
Acc.
.55
593.3
.05
662.2
.71
593.2
.66
656.9
Table 1: Accuracy on the occupation classifi- cation task. In In Context experiments, the target entity's biography is prefixed to the prompt, while in No context only the entity's name is provided. In both settings, REMEDI leads GPT-J to generate fluent and more accu- rate text.
formed in layer 12 (this and other hyperparameters discussed in Appendix C). We evaluate factual accuracy and fluency before and after applying REMEDI on 5000 test examples. Accuracy is evaluated by measuring how often the highest occupation is the true one, and fluency using the same n-gram entropy measure as Meng et al. (2022a).
Crucially, we train REMEDI only out-of-context:: choosing the initial text x to consist only of the entity's 's name. However, as described below, we evaluate its ability to control model behavior both in-context (with additional text provided) and out-of-context (on inputs similar to the training data).
Results Results are shown in the left portion (In Context) of Table 1, which reports GPT-'''s factual accuracy and fluency before and after applying REMEDI. REMEDI increases GPT-J's accuracy by over 15% on held-out (entity, attribute) pairs, showing that representations produced by REMEDI more often encode the desired attribute. REMEDI also preserves the fluency of the generated text. We find similar improvements in other models (Appendix F).
5

---

Published as a conference paper at COLM 2024
Generic Entity
Attribute
In-Context Generation
REMEDI Generation
Emily
She specializes pediatric trauma of the upper and lower extremities, and lower extremity reconstruction in- cluding treatment of limb length discrepancy...
Emily physical therapist, and has been practicing for over 10 years. She has B.S. in Physical Therapy from the University of North Car- olina at Chapel Hill, and M.S. in Physical Therapy from University of North Carolina at Chapel Hill.
occupation
Emily has occupation of surgron. surgery the University of Minnesota Medical School.
professor
Jamie
He specializes in the training of athletes, most specifically golfers.
Jamie has occupation of professional golfer. He has played on the PGA Tour and the Nation-
Jamie has the occupation of professor dietetics. He has bachelor's 's degree in dietetics from University of Minnesota and master's 's degree in nutrition from the University of Miami.
Tour.
Famous Entity
Leonhard Euler
domain of activity opera
Leonhard Fuler is the most pro- lific mathematician of the 18th century. He is best known for his work in number theory, algebra, geumetry, and analysis.
Leonhard Euler is a compuser of opern. He was born Venice, Italy, and studied at the Ac- cademia Santa Cecilia in Rome.
Kingdom of the Netherlands
is located in Antarctica
/ Kingdom of the Netherlands is
/ Kingdom of the Netherlands is country in Antarctica. It is the world's 's third-largest country by land area, after Russia and China.
sovereign state in Antarctica, a member the Antarctic Treaty System. It is the only state in permanent the world that paaaaappeaaaa population in Antaretica.
Table 2: Examples of GPT-J generations when the attribute specified in the textual prompt (In-Context Generation) or inserted via REMEDI. For both generic and famous entities, REMEDI causes GPT-J to respect the attribute more often than when provided in context.
We contextualize these results by evaluating model behavior when the LM has no lextual context (i.e. no initial biographical sentence). Here, the base LM has information about entities' occupations, and obtains near-chance factual accuracy. However, inserting REMEDI's representations into the LM causes it to generate fluent text consistent with the edit, showing that REMEDI can not only enforce coherence with a textual context, but replace textual prompting by inserting information directly into entity representations. REMEDI is slightly more effective at in-context editing than out-of-context editing, despite being trained only out-of-context. The last column of Table shows examples of in-context generation.
4.2 Editing Factual Associations
We next show REMEDI can be used to overwrite background knowledge about entities with new and even contradictory facts. As shown in Fig. 1, when LMs are prompted with text like To cross London Bridge, one should travel to, they often complete it with true or plausible text like to the South Bank [in London]. This knowlèdge is derived from training data (which contains many co-occurrences of the strings London Bridge and South Bank), and is difficult to override: when contradictory information is provided in context, LMs sometimes ignore it. Most current work updates LMs by altering their parameters (De Cao et al., 2021; Mitchell et al., 2022; Dai et al., 2022; Meng et al., 2022b). They all share the limitation of changing the behavior of the LM globally: users cannot choose when to apply edits. In existing methods, edits often bleed into closely related but distinct entities (Meng et al., 2022a). Because REMEDI operates directly on entity representations at runtime, it applies changes only to the entity of interest at the moment of use.
We evaluate REMEDI on the COUNTERFACT benchmark from Meng et al. (2022a), which consists of (subject, relation, old value, new value) tuples-e.g. (Megan Rapinoe, plays sport, soccer, tennis) and measures LMs' ability generate natural text consistent with the new fact.
Method We train REMEDI on subset of 5000 examples from COUNTERFACT and evaluate on held-out subset of 5000. As before, we take Mentity to be the last token of the entity

---

Published as a conference paper at COLM 2024
mention (which appears at the beginning of COUNTERFACT examples) and hattr to be the average representation of the new fact in context. For example, we pass (Megan Rapinoe plays the sport of soccer) to the LM and compute hattr from the underlined tokens. This textual context is akin to the biographical sentence used to compute haur in the previous section. We use all three loss terms from Eq. (6) and apply edits in layer 1; see Appendices B and C for other hyperparameters and implementation details. As above, we train REMEDI without textual context, with inputs x consisting of entity names alone.
Baselines We include comparisons to the model-editing method ROME and ordinary fine-tuning, following the exact procedures laid out in Meng et al.. However, our primary baseline one in which the new factual information prepended to the prompt. In all other methods, the language model is only given a prompt with no context about the fact. We additionally include a baseline in which we find-and-replace the entity with one that shares the target attribute (e.g,, replacing Versace headquarters with Harrods). This provides a realistic upper bound on LM consistency and fluency after editing (because the LM has not been modified or conditioned on -ut-of-distribution text).
Metrics We follow the evalua- tion schema from Meng et al. and track the core metrics reported there. Efficacy measures how of- ten PLM(()g) > PLM(4prior when the intervention is applied to a held out prompt that paraphrases the target attribute. Neighbor- hood score measures how often the LM's predictions about sim- ilar but distinct entities change. Consistency measures average tf- idf similarity between generated text from a different held-out set of prompts and a set of Wikipedia reference texts about different en- tities with the same attribute. Flu- ency is the average bi- and tri- gram entropy of generated text, designed to be low for degenerate or repetitive outputs. Essence captures how much the edited entity is still "itself" according to the model (is London Bridge still bridge?). Formally, it measures tf-idf similarity between the model's generations before and after the intervention given the prompt: {Entity] is
Rep. Edit
Eff.
Nbr.
Cons.
Fl. 1
Ess. f
Prefix Replace REMEDI
80.2
100.0
21.6
591.4
40.5
79.9
100.0
33.0
613.3
7.5
98.2
100.0
33.6
598.8
24.8
Model Edit
FT ROME
100.0
10.6
23.5
381.3
28.6
100.0
79.1
43.0
620.1
27.0
Table 3: Results from the COUNTERFACT benchmark. REMEDI comparably effective (Efficacy, Consistency) to model editing methods at eliciting generations con- sistent with the target attribute, and is more effective than prefixing the prompt with the new fact. Unlike model-editing methods, REMEDI does not influence gen- erations about different entities (Neighborhood) avoids degenerate output (Fluency) and preserves most origi- nal features of the entity (Essence).
Results Table 3 shows metrics for REMEDI and baselines. Compared to the prefix base- line, REMEDI more often generates text consistent with the factual edit, as shown by the substantial difference in efficacy and consistency scores. The base LM incorporates textual prompt information 80.2% of the time, while REMEDI-based prompting incorporates new information 98.2% of the time. This performance comes some cost to the essence of the entity, likely because the original fact is strongly associated with other properties of the entity. Table 2 shows several examples; for example, when Leonhard Euler edited to work on opera, LM output describes him as being born in Venice Italy. While this output has lost some of Euler's 's identity as a Swiss academic, also respects implicit correlations between facts (e.g. that opera is more strongly associated with Italy than Switzerland). Appendix D contains a complete additional set of experiments on a word-redefinition task that studies how REMEDI models these correlations in more detail. Further analysis of REMEDI's factual editing performance is provided in Appendix F. Results for other models are in Appendix
REMEDI is as effective and substantially less destructive than fine-tuning, While ROME produces slightly more consistent generations with respect to the updated fact, it comes at
Thii is called efficacy score (ES) in Meng et al.
7

---

Published as a conference paper at COLM 2024
the cost of altering neighboring entities: 221% of the time, ROME causes facts about related entities to change, whereas REMEDI never causes such failures.
Detecting Errors
Next, we show that REMEDI can be used as model eoaluation tool, automatically charac- terizing when (un-modified) LMs have successfully acquired background or contextual knowledge, A core challenge when deploying language models is that is difficult to automatically detect when they exhibit the failures shown in Fig. 1. Some work rddresses this challenge by calibrating LM predictive distributions et al., 2020a), or training auxiliary models to reject bad samples (Cohen et al., 2022). REMEDI offers a new approach to detecting when LMs will fail to integrate information from context: inspecting their representations for the information that REMEDI would add. This approach is related to a method by Burns et al. (2022), which finds implicit LM encodings of veracity. REMEDI identifies cases where even LM-internal states encode errors.
better reflect veracity (Jiang
Method Suppose we have a prompt that queries an LM for a fact: To cross London Bridge, one should travel to .... Under the hypothesis from Section 2, the LM should answer this question exactly when the representation of London Bridge is already aligned with an encoding of the fact is located in London. Taking Nattr to be the average representation from is located in London, we can use REMEDI to compute such an encoding:
dattr F(0, hatr)) Whattr bb.
We may then quantify how strongly an LM encodes the fact by computing:
Hentiydattr = hentiyy (Whattr + b),
(8)
analogously to the knowledge probe in Eq. (2). Given true attribute of the London Bridge (located in London), and alternative (or "distractor") attributes (located in Arizona), we can compute directions ddist for distractors, and predict that an LM will err if its representation of London Bridge is more aligned with any distractor than the input:
I Ventitydats 3 N M dattr
(9)
Given the success of REMEDI as an editor for entity representations provided with and with- out textual context, we might further expect REMEDI-based probing to detect contradictions with input text as well as training data. Below, we show how to use REMEDI to identify errors of both types. Limitations of this approach are discussed in Appendix A.
5.1 Detecting Errors in Prior Knowledge
Setup and method We revisit the COUNTERFACT dataset used in Section 4, and use the same REMEDI model to produce attribute encodings. We compute target attribute encodings datr from the dataset's ground-truth facts (is located in London), and distractor attribute encodings ddist from the dataset's s counterfactuals (is located in Arizona). If Eq. (9) is satisfied, we predict that the LM will generate an incorrect output.
Baselines and Controls We compare REMEDI to several baselines, upper bounds, and controls (Hewitt & Liang, 2019). The identity encoding model takes d and datte to be the untransformed LM encodings of the two attributes in question. The fact probe is the model of Li al. (2021), trained to predict ground truth facts from LM encodings. Shortcut is version of the same model trained to predict the model's ss own preferred outputs from its hidden states, analogous to Belrose et al. (2023) (though without that work's additional use of the model's own unembedding layer). We apply REMEDI to a random model: a randomly initialized GPT-J with an editor trained as above (to characterize whether our evaluation surfaces factual knowledge acquired during pre-training). Finally, we contextualize these results with a supervised error skyline model, which is trained to predict whether a model will fail in context without identifying any specific output incorrect. (This model, which is similar to the approach of Mielke et al., 2022, must be trained on model outputs annotated with correctness information, and is not directly comparable REMEDI.)
8

---

Published as a conference paper at COLM 2024
Results The Prior column of Table shows results for the fac- tual error detection task. We re- port both the Fi-measure and coefficient (Matthews, 1975; Chicco & Jurman, 2020) to cap- ture how well each method predicts true negatives (model will produce correct outputs) as opposed to just true positives (model will fail). In the prior knowledge setting, REMEDI out- performs all methods (in F1 and Φ) except for the shortcut model, which is trained directly for this task. Even when trained to perform representation edit- ing, REMEDI finds directions that align with models' unedited rep- resentation of true facts, and these directions are specific to trained LMs.
Prior
Contextual
Fi
F1
Φ
Identity encoding
.34
.34
.08
Fact probe (Eq. 2; Li et al., 2021)
.33
.21
.38
.18
Shortcut (cf. Belrose et al., 2023)
.53
.43
.40
.21
REMEDI
.39
.26
.42
.24
Control: Random model
.54
.09
.51
.04
Skyline: Superoised errors
.94
.93
.94
.93
Table 4: Fl scores and & coefficients for predicting LM behavior on the COUNTERFACT dataset. In the Prior condition, the LM is prompted to predict property of an entity. In the Context condition, the prompt includes additional information, and REMEDI predicts whether the LM's s preferred completion will contradict this context. Here REMEDI trained as in Section 2, to perform editing in non-contextual sentences only. Nevertheless, when used as a probe, REMEDI encodings detect errors more accurately than existing knowledge probing methods. Results for other models are in Appendix E.
5.2 Predicting Errors in Context
Setup and method We next use REMEDI to detect failures to incorporate new information provided part of model's textual input: for example, The London Bridge is located in Arizona. To cross the London Bridge, one should travel to. We use the new information (is located in Arizona) to compute the target attribute direction dattr, and the prior fact (is localed in London) for the reference ddist- We predict the language model will fail to incorporate
the context (will rank fprior London higher than figt Arizona) if Eq. (9) holds. Results in this section use the same REMEDI model as in Section 4-which optimizes attribute encodings to influence model generation on sentences without additional textual context. These experiments thus measure the extent to which REMEDI encodings characterize both background knowledge and information provided in-context.
Results Applied detection of contextual errors, REMEDI outperforms all baselines, including the shortcut model (Contextual column of Table 4). REMEDI thus generalizes across knowledge sources, discovering common encodings of background and contextual knowledge. We emphasize that this detection procedure is not extremely accurate, and model directly supervised with information about LM errors performs significantly better. However, these results show that REMEDI encodings (learned out-of-context) are, to non-trivial extent, aligned with LMs' representations of knowledge provided in-context.
Conclusions
Factual knowledge in neural language models can be interpreted and controlled by applying local transformations to contextual representations of entity mentions and other nouns. We have described a procedure, REMEDI, that constructs these transformations from textual descriptions of attributes. By amplifying fact's encoding, we can force LMs to generate text consistent with that fact (even when a textual prompt fails to do so). Similarly, by inspecting models' representations, we can sometimes detect the absence of a fact encoding and predict that the language model will err. While not without limitations (Appendix A), our findings suggest new path toward controlling LMs: instead of providing textual context or instructions, models may be controlled by directly intervening in their internal representations.

---

Published as a conference paper at COLM 2024
Ethical Considerations
As language models are deployed for increasingly complex and high-stakes tasks, the ability to control their generations promises to be both a boon and risk. Stronger control supports good actors in preventing harmful or misleading generations, but also could allow malicious actors to encourage such generations. Ultimately, we believe LMs pose greater risk uncontrolled, where incoherent or factually incorrect generations will directly reach users in trusted applications. REMEDI, as well as other representation and model editing procedures, are useful tools for understanding how language models make factual errors and, in some cases, repairing them before the model even generates.
Reproducibility Statement
All code and data used in this paper, including the REMEDI python library and the code used to generate figures in this paper, will be made publicly available upon publication. Experiment details are described at the beginnings of Sections 4 and 5. In addition, describe our dataset preprocessing procedures in Appendix B and our hyperparameter sweeps in Appendix C. We ran all experiments on workstations with 80GB NVIDIA A100 GPUs or 32GB Tesla V100 GPUs using the HuggingFace Transformers library (Wolf et al., 2019) implemented in Py PyTorch (Paszke et al., 2019).
Acknowledgements
We thank David Bau for helpful discussions. EH and JA gratefully acknowledge support from Sony Faculty Innovation Award, a grant from Liberty Mutual through the MIT Quest for Intelligence and a gift from the Open Philanthropy Foundation. BZL additionally supported by an National Defense Science and Engineering Graduate Fellowship.
References
Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. Can language models encode perceptual structure without grounding? case study color. Proceedings of the 25th Conference on Compulational Natural Language Learning, pp. 109-132, Online, November 2021. Association for Computational Linguistics. doi: 10.18653 v1 202022.co1ll11.......1 URL https: h/acaantooo/ll org/22211con111111 conll-1.
Ekin Akyürek, Tolga Bolukbasi, Frederick Liu, Binbin Xiong, Ian Tenney, Jacob Andreas, and Kelvin Guu. Tracing knowledge in language models back to the training data, 2022. 482.
URL https://arxiv. http:/iv..rggab/////eee
Yonatan Belinkov and James Glass. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49-72, 2019. doi: 10.1162 tacla _00254.. URL.https://cclanttoolo. org//1gg///////// 1004.
Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. ar Xiv preprint ar arXiv:2303.081 001121 2023.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Majon and Shmarggret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, pp. 610-623, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383097. 1145/3442188 3445922.
doi: 041145/3442188.345553
URL https://doi..o
Ali Borji. A categorical archive of chatgpt failures, 2023. URL htthh: //t::///a/x/vv//i h/tp:////////a...oo//////////// 03494.
10

---

Published as a conference paper at COLM 2024
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vosse Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language mod- els are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL ptattts::/prrceeedeoo 1457cdd6bfcb49674 81re4411bbbbaccccco 42ta
/eepr/ps.cc/pppe//22///iiii
pdf.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. ArXiV, 2022.
Davide Chicco and Giuseppe Jurman. The advantages of the matthews correlation coefficient (MCC) over fl score and accuracy in binary classification evaluation. BMC Genomics, 21(1), January 2020. doi: 10.1186/ 2116///81110---611-777 URL https://doi..rr//1 s12864-019-6413-1
Aaron Daniel Cohen, Adam Roberts, Alejandra Molina, Alena Butryna, Alicia Jin, Apoorv Kulshreshtha, Ben Hutchinson, Ben Zevenbergen, Blaise Hilary Aguera-Areas, Chung ching Chang, Claire Cui, Cosmo Du, Daniel De Freitas Adiwardana, Dehao Chen, Dmitry (Dima) Lepikhin, Ed H. Chi, Erin Hoffman-John, Heng-Tze Cheng, Hongrae Lee, Igor Krivokon, James Qin, Jamie Hall, Joe Fenton, Johnny Soraker, Kathy Meier- Hellstern, Kristen Olson, Lora Mois Aroyo, Maarten Paul Bosma, Marc Joseph Pickett, Marcelo Amorim Menegali, Marian Croak, Mark Díaz, Matthew Lamm, Maxim Krikun, Meredith Ringel Morris, Noam Shazeer, Quoc V. Le, Rachel Bernstein, Ravi Rajakumar, Ray Kurzweil, Romal Thoppilan, Steven Zheng, Taylor Bos, Toju Duke, Tulsee Doshi, Vincent Y. Zhao, Vinodkumar Prabhakaran, Will Rusch, YaGuang Li, Yanping Huang, Yanqi Zhou, Yuanzhong Xu, and Zhifeng Chen. Lamda: Language models for dialog applications. In arv.v. 2022.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Ba- roni. What you can cram into a single $&.#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeling of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2126-2136, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P1-111 11988
URL
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational S (Volume 1: Long Papers), pp. 8493-8502, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022. acl-long.581 58.. URL https: httpss://acaanthology orgpoo22..c.....oo acl.:.2..l11..... acl-long....... long.58. 581.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. In International Conference on Learning Representations, 2020. met/forum?ion Hd= edEyBKDS.
URL https://openrreiiew http://ppenreiiieeeneee/.orrrroe
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT '19, pp. 120-128, New York, NY, USA, 2019. Association for Computing Machinery. ISBN URL http::///00..rrg/1.1144//////
9781450361255. doi: 10.1145/3287560.228888 3287572.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6491-6506, Online and Punta Cana, Dominican Republic, November 2021. Association
11

---

Published as a conference paper at COLM 2024
for Computational Linguistics. doi: 10. 00.16663/112021.emmlpp- //aclanthology.oo org/ eenllp-mmnllplmimmeee 522.
URL https:
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understandling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computatioonll Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/ N19-1423. URL https: / http//allaatttt http///aaaaaatooooyoooooo org/N19-1423,
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing:: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160-175, 2021. doi:
URL https://aclanthoolog org/
2021, t.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling, ar arviv preprint 1xXi:2101.00022 2020.
Gabriel Grand, Idan Asher Blank, Francisco Pereira, and Evelina Fedorenko. Semantic projection: recovering human knowledge of multiple, distinct object features from word embeddings. CoRR, (bbb/.1110.1111,, 2018. URL httptt org/ abs/ 01241.
Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. More than you've 've asked for: A comprehensive analysis of novel prompt injection threats to application-integran //arxiv.org www.aaa/..... 12173.
large language models, 2023. URL https:
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 107-112, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/
v1/N18-2017. URL https:://acaattholo.oo /rgpe///nnnnoooooooo///1----11 201777
Peter Hase, Mona T. Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Tyer. Do language models have beliefs? methods for detecting, updating, and visualízing model beliefs. CoRR, abs/2211.13654 2021. URL
https://arxiv orgg/bs//21111 13654.
John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2733- 2743, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1222
org/D19-1275
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How can we know when language models know? CoRR, abs/ 2012.00955, 2020a. URL https://arxiv http://axxvvvoorrraaa/ org/abs/ 2012.00955.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423- 438, 2020b.
doi: 10.1162 1a1112/taac... tttp//lllll.000/11 000244 URL https: org/ 2020. tacl-1.28.
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation, 2019. URL https://arxiv https::/arxiivorr//bb//// 1909. 05858.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 785-794, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/ (0.8653/vl/DDD v1/D17-1082 URL https://acl l/atthoo//
12

---

Published as a conference paper at COLM 2024
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/ /2021 wtt1///22eemm////11/ mennlp-main.243. URL
org/ 2221. momnpp-main... 243.
Belinda Z. Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models, In Proceedings of the 59th Annual Meeting of the Associa- fion for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1813-1827, Online, August 2021. As- sociation for Computational Linguistics. doi: 10.18653/v2 /2021 2021.acl-loog 143. URL
nttps://aclanthol..0 org/2221 acl-long. 143.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Pp. 4582-458555 4597, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ /tt////111121/1////111 122/222.aclloong.5555.... URL http:///aaaaaaatoa.aa.//oo wrtpllogg..r///235 org/2021 acl- 353.
Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2017.
B.W. Matthews. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica Biophysica Acta (BBA)- Protein Structure, 405(2):442-451, October 1975. doi: 90109-9.
URL https:://ooiii org/10. 100.111111/0000775557777
Ken McRae, George S. Cree, Mark S. Seidenberg, and Chris Mcnorgan. Semantic feature production norms for large set of living and nonliving things. Behavior Research Methods, 37(4):547-559 November 2005. ISSN 1554-3528. doi:
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 36, 2022a.
Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass editing memory in a transformer. ar Xiv preprint ar Xii::2210.07229, 2022b.
Sabrina J Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. Reducing conversational agents' overconfidence through línguistic calibration. Transactions of the Association for Computational Linguistics, 10:857-872, 2022.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning, Fast model editing at scale. In International Conference on Learning Representations, 2022.
URL
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An im- perative style, high-performen processing systems, 32, 2019.
deep learning library. Advances in neural information
Hao Peng, Xiaozhi Wang, Shengding Hu, Hailong Jin, Lei Hou, Juanzi Li, Zhiyuan Liu, and Qun Liu. Copen: Probing conceptual knowledge in pre-trained language models. In Proceedings of EMNLP, 2022.
Fábio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for language models, 2022. URL https: http://awxxvv.oooabb/////ss 09527.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2463- 2473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653 1118665/v11D1111555 URL https: http///lllaatolllooor///////////
13

---

Published as a conference paper at COLM 2024
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, PP. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/ (0118633/11/D161111 /1/1///11/1///A URL https://aclantthol..g hww.hooloooo.og///1111 org/ 1264.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Amnual Meeting of the Association for Computational Linguistics, pp. 7237-7256, Online, July 2020. Association for Computational Linguistics. doi: 10.18653 /v/53/v//// / mod-mmain.44.. URL
. org/ 2020.0000... aal-maiin.64
Abhilasha Ravichander, Yonatan Belinkov, and Eduard Hovy. Probing the probing paradigm: Does probing accuracy entail task relevance? In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 3363- 3377, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v1 2021.eadlmain.299 295. URL https::/aclanteee www.lloyyy..og..goeeeeee org/2021. eacl- maalmmmin..99 295.
Siva Reddy, Danqi Chen, and Christopher D. Manning. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266, 2019. doi: 10.1162 t0026. URL https: /aaps::/aaaaannee htt///aacaantoooolo/o/oo .acpea...gggg///... 0rg9019-1016.
Kyle Richardson and Ashish Sabharwal. What does my QA model know? devising con- trolled probes using expert knowledge. Transactions of the Association for Computational Linguisfics, 85872-588, 2020. doi: 10.1162 tacl a 00331. URL https: htttp:://aclaatholloyy org/ 2020. tacl-1.37.
Alexis Ross, Tongshuang Wu, Hao Peng, Matthew Peters, and Matt Gardner. Tailor: Generating and perturbing text with semantic controls. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3194-3213, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653 / /12/0022.clloooog222222 228. URL https://aaaee http://aaaaaatoooy...ooo .rgh wwtpg//222..co long. 228.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brían Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Cur- ran Associates, Inc., 2017. URL https: httppe//////oceiinnss. www.oeeeinnnneeersssccccooaoaeee metrrpps.ccpppprr////////
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A Billion Parameter Autoregressive Lan- guage Model.
May 2021.
14

---

Published as a conference paper at COLM 2024
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. arXio preprint arXi::1910.03777,, 2019.
A Limitations
Our goal in this work has been to demonstrate the expressive power of REMEDI's represen- tation edits. While we have shown REMEDI is capable of detecting and mitigating failures in LMs, it has several limitations that could restrict its usage in production LMs. The foremost is that REMEDI's linear editing functions must be learned, which means users must construct or have access to in-domain training data of the format considered here (each sample has a prompt, entity, attribute, and farget word). Similarly, using REMEDI to detect failures of context integration or to detect the absence of prior knowledge requires users to know the correct attribute a priori and to have access to distractor attribute for comparison; neither may be available in practice. Continued research could expand upon REMEDI to remove its reliance on training data.
Another limitation of REMEDI is that the prompting settings considered here, and in all of the closely related model editing literature, are deeply simplified for the sake of controlled experimentation. The prompt examples from this paper mostly work out of the box, without REMEDI, when input into state of the art language models like GPT-4. However, the failure modes we study-factual mistakes and ignoring contextual information are well documented even the most performant language arise in subtler ways, from more complex prompts, than failures in standard benchmarks.
(Borji, 2023). The failures simply
B Dataset Preprocessing
In Section 4, we evaluate REMEDI on two dataset (and a third in Appendix D. Here we detail how they are preprocessed and formatted.
For each record, we use the first paraphrase prompt with the post-edit target object appended to it as the context. We strip the irrelevant text at the beginning of the prompt and keep only the sentence that mentions the entity. We take the attribute to be every token after the entíty in the context. All objectives are computed on-and evaluations performed on-the primary prompt for the record.
COUNTERFACT
Bias in Bios For each record, we take the second sentence in the bio longer than three words to be the context. If the sentence does not mention the entity, we prepend the phrase About /Entityl: to it. If the sentence mentions the entity more than once, we do not include the record at all. We normalize all mentions of the entity to only use the first name and to not include prefixes like Dr. We set the prompt to be [Entity] has the occupation of. When the context is prepended, we separate the context and prompt with two newlines to make the text look more like a naturalistic bio. The target word is the person's 's normalized occupation. Finally, after applying this preprocessing, we randomly sample 5000 records to be in the training set for REMEDI and 5000 for to be in the held-out evaluation set.
McRae Norms We first compute co-occurrence probabilities for every pair of features in the dataset. For each concept e (e.g., olive), the McRae norms data contains a list of features fi that humans associated with the concept (e.g., is green, or is edible). The data additionally provides a probability p(fi c) representing how many people out of thirty ascribed the feature to the concept. Using this data, we sample pairs of features fi and f2 that co-occur for at least one concept and estimate their co-occurrence probability as follows:
p(f2 1 fi) = f(f
1
p(f) p(f) Erla c)p(A c))((c) =
N₂ Ep(fa c))(())) (c) (10)
6The first sentence often explicitly states the person's occupation.
15

---

Published as a conference paper at COLM 2024
REMEDI Score across Layers (CounterFact)
REMEDI Classification Fl (CounterFact)
1.0
0.4
0.2
E
0.5
0.0
0
4
8
12
16
20
24
8
12
16
20
24
Layer
Layer
REMEDI Score across Layers (Bios)
REMEDI LM Human Correlation (McRae)
MPLM Pa)
Scare
1.0
0.2
0.5
0.0
0.0
16
24
20
24
Layer
Layer
Figure 3: Left Column: Harmonic mean of all the generation quality metrics from Section 4 after applying REMEDI at each layer of GPT-J on a subset of 1000 samples from each dataset. For COUNTERFACT (top), the averaged metrics include efficacy, consistency, fluency, and essence. For Bias in Bios (bottom), it includes accuracy and fluency. Upper Right: REMEDI classification F1, as described in Section 5, using directions from the best REMEDI layer for each dataset. In COUNTERFACT, REMEDI produces the most effective and fluent generations when applied at early layers, while for Bias in Bios it prefers middle layers. Upper Right: classification is most precise when applied to layers after the edit layer. Lower Right: Post- edit human-LM correlation, as defined in Appendix D, when applying REMEDI at different layers. REMEDI works best at earlier layers.
where the sum is over concepts and where Nú, is the number of concepts for which at least one person mentioned fi. Notice that we assume fi and fz are conditionally independent given c, and that p(c) is uniform.
We use these human-derived probabilities in two ways. First, when we compute the correlation between PLM and PH, we take PH to be p(fi f2) when evaluating against correlated features and p(fi c) when evaluating against the original features of the concept. Second, we use p(fi f2) to filter the set of candidate feature pairs, including only those pairs with co-occurrence probability greater than .1.
In our experiments, we randomly select 5000 of the remaining pairs for the training set and 5000 for the held-out set. For each sampled pair, randomly select concept that does not have either feature, and choose one feature to be the context and the other to be the test prompt. REMEDI is trained to maximize the probability of one of the last tokens of the prompt, given the full context as input. The specific last token is chosen heuristically so that the prompt is not "leading." " For example, if the prompted feature used for eating, then the target word is eating, while if the prompted feature is grows on trees, then the target word is grows. See the code release for the full implementation.
C Training Editors
For both the COUNTERFACT and Bias in Bios datasets, we train F using Eq. (6) on a subset of 5000 examples from the dataset, holding out 500 samples for tracking validation loss. For COUNTERFACT, we set A₁ = 1 and 2 = 10. For Bias in Bios and McRae Norms, we set A = 0 and do not use the Cprior term. We optimize using AdamW (Loshchilov & Hutter, 2017) with a learning rate of .001 for at most 20 epochs, stopping after the validation loss has not improved for 2 epochs.
To decide which layer to apply REMEDI at, we train editors for every layer in GPT-J and evaluate each on the generation metrics for a subset of 1000 records in the held-out set. For metrics requiring open-ended generation, we use greedy decoding for these sweeps and top-k sampling (k = 5) in the final evaluations. Fig. 3 (left, bottom) plots the harmonic mean of all generation metrics used in each task (listed in corresponding subsections of Section 4).
16

---

Published as a conference paper at COLM 2024
Effect REMEDI( for chopping wood)
Correlated Features (*1)
Original Peatures Edited Concept (f")
Features
Froat cen
Por ea zu
la hard!
Stack
porr a Crom
bus auth Lockut
found io Sndrromt
used provecton)
caiiy lost
Figure 4: Change in LM log-p pag-robabblity for different feature strings after using REMEDI to add the feature is used for chopping wood to seven different concepts. Each point corresponds to feature and is bucketed by whether it is correlated with the added feature (left), is original feature of the concept under edit (middle), or is random (right). Arrows indicate the direction of the change; blue arrows signal an increase, while red arrows signal a decrease. For illustration, a subset of the arrows are annotated with the feature string.
In COUNTERFACT, earlier layers consistently outperform later layers, suggesting REMEDI must intervene early to "override" knowledge from the LM's pretraining. By contrast, for Bias in Bios and McRae Norms, REMEDI's performance relatively flat across early and middle layers. Based on these plots, we chose to apply REMEDI at layer 1 for COUNTERFACT, and layer 12 for Bias in Bios and McRae.
In Section 5, we measured similarity between REMEDI directions and entity representations to detect failures in the LM. To decide which layer take the entity representation from, we compute classification F1 for each layer. Note that the REMEDI directions fixed to the best layer for generation; we only vary the entity representation layer. Results are shown in Fig. 3 (upper right). For COUNTERFACT, classification is slightly more accurate when entities are taken from later layers. For Bias in Bios, middle layers are best.
D Redefining Concepts
In our final set of generation experiments, we use REMEDI to edit basic noun concepts (like olive or airplane) and change their definitions. Noun concepts are typically defined by the set of features language users associate with them: olives can be green and often appear in salads; airplanes largely made of metal and can fly; and spiders have eight legs and spin webs.
Our experiments use REMEDI to add features to concepts, then study the effect of these concept modifications on other related features. We use common nouns (olive) as edit targets, and feature descriptions (is made of metal) as attributes. Properties like is made of metal, is hard, and is shiny exist in complex network of entailment and correlation relations, and we are interested in characterizing whether REMEDI respects these associations (e-g. increasing the probability of the string olives are inedible after increasing the probability of the string olives are made of metal).
Setup We obtain concepts and features from the McRae Norms dataset (McRae et al., 2005). This dataset contains 541 concepts, 2526 features, and information about the frequency with which each feature was described prototypical of each concept by human raters. We construct a dataset containing 10k entries, split evenly into train and test sets, where each entry consists of concept c, a list of original features f(() for the concept, target feature to add f*, and list of features f(e) that are correlated with the new feature. Details about data and hyperparameters are in Appendices B and C.
Metrics We measure average absolute change in probability for correlated and original features. If f is any held out feature string (f(0) or fle)), we define absolute change as:
Af c,f") P c,f") P (c.
(11)
17

---

Published as a conference paper at COLM 2024
Correlated
Original
Rand.
Method
APIm
ΑPIM
APım
No Edit
.11
.26
Prefix
0.4
(0.7)
.16
0.0
(1.7)
.25
0.0 (0.0)
REMEDI
7.1 (5.2)
.29
0.5 (3.6)
.19
0.2 (0.9)
Table 5: Comparison between REMEDI and a prefix baseline for adding new features to concepts from McRae et al. (2005). APLM is the mean (SD) of the absolute change in LM probability assigned to feature strings, scaled by 100. is shorthand for (PLM, PH), the correlation between the post-intervention LM probabilities for features and their human- derived counterparts. Compared to prefixing, REMEDI causes large increase in PLM for all correlated features, as well as modest changes to original features in either direction. On random, unrelated features, both methods have little effect. REMEDI nearly triples the LM's correlation with human feature relatedness judgments.
where PLM(-) denotes the probability that the LM assigns to f conditioned on c as a prompt and with f* added to the concept via textual prompting or via REMEDI. We additionally measure the correlation between LM probabilities and human-derived probabilities PH(f) for held-out features, which we denote r "PPM, PH). For original features, we compute PH (f())) as the proportion of human annotators who described f(o) as a prototypical feature of the concept being edited. For correlated features, we compute pr(((()) as the co-occurrence probability with the feature being inserted.
Results Table compares REMEDI to the prefix baseline where the new attribute (e.g. An olive is made of metal) is prepended to the prompt. Using REMEDI results in much stronger effect than prefixing: correlated features see an order of magnitude larger increase in probability and become substantially more correlated with the human-derived feature co-occurrence probabilities. This suggests that REMEDI preserves the correlates of added attributes: an olive, now made of metal, more likely to be shiny.
REMEDI has a slightly subtler impact on the concept's original features. The near-zero mean and large standard deviation highlight that some original features are promoted under REMEDI while others are suppressed, likely because they conflict with the added feature (e.g. olives cannot be both made of metal and edible). This is further reflected in the decrease of (PLM, PH): :: the language model's post-edit distribution over a concept's 's original features less resembles the human distribution. Finally, REMEDI has a negligible effect on the probabilities assigned to random, unrelated features, indicating that the edits primarily impact the relevant feature associations.
Figure 4 provides a concrete example of REMEDI's effect on PLM when adding the is used for chopping wood feature. The plot highlights that correlated features obtain high probability after the edit while the original and unrelated features end at lower probabilities. Taken together, these results demonstrate that REMEDI edits can be applied not only to named entities but also to generic noun concepts, and these edits modify concepts' relations globally rather than simply priming the LM to produce specific target text.
E Results in Other Models
We run the evaluations of REMEDI from Sections and 5 on two additional models: GPT2-XL. (Radford et al., 2019), which has fewer parameters than GPT-J (1B vs. 7B) and Llama-2-13b (Touvron et al., 2023), which has more (13B vs. 7B). We select layers to perform editing and classification layers using the same procedures as before. Results are in Tables 6, 7, and 8.
In general, the same trends observed in GPT-J hold across GPT2-XL and Llamb-2-13b, with greater effectiveness model size increases. For generation tasks, REMEDI exerts more effective control than standard prompting. For error classification tasks, REMEDI transfers effectively from non-contextual training to contextual classification in all models.
18

---

Published as a conference paper at COLM 2024
Model
Method
Ctx Acc.
Ctx Flu.
No-Ctx Ace.
No-Ctx Flu.
GPT2-XL
LM only
0.079
560.1
0.01
646.6
REMEDI
0.084
539.8
0.27
554.7
Llama-2-13b
LM only
.71
491.3
.04
625.7
REMEDI
.72
457.1
.64
599.5
Table 6: Results on the error correction task of Section 4.1 for two different models. Mirrors Table 1.
Model
Approach
Eff
Nbr
Cons
Flu
Ess
GPT2-XL
Prefix
844
1.0
.24
594.1
.39
Replace
.75
1.0
.33
609.3
.08
REMEDI
.97
1.0
.32
597.0
.28
Llama-2-13b
Prefix
.84
1.0
.17
557.0
.41
Replace
.86
1.0
.30
591.8
.06
REMEDI
.98
1.0
.25
550.5
.29
Table 7: Results on the factual editing task of Section 4.2 for two different models. Mirrors Table 3.
Table 6 reveals several qualitative differences between models. In particular, GPT2-XL is extremely ineffective at following the prompt on the BiasInBios task (Table 6 Ctx Acc.), while REMEDI, applied without textual context, is substantially better steering model behavior in such a small model. REMEDI is less effective when a textual context is also provided, but still improves slightly upon the baseline. By contrast, Llama-2-13b consistently follows the prompt at baseline, and exhibits relatively low error rate to begin with. REMEDI, applied in context, improves accuracy by small amount, but by much less than when the baseline LM is ineffective.
F
Analyzing REMEDI Edits
F1
Failure Modes
Table shows examples of REMEDI's failure modes, taken from the evaluations of Section 4.2. While Tables 1 and show that REMEDI is effective at causing the LM to generate text consistent with the attribute, the act of editing the LM's representations can occasionally lead to disfluent or incorrect generations. In generic entities, these cases primarily involve REMEDI failing to insert the attribute, or only inserting part of it. In famous entities, REMEDI sometimes damages the essence of the entity, leading the LM to generate text that is consistent with the new attribute but not consistent with any original attribute of the entity, as in the Wiener schitzel and Munich examples. REMEDI can also cause unrelated facts to change, such as the airtime of My Name Earl in the bottom row.
Some of these errors might originate from the model itself. We observe disfluent, repeating generations even when we do not apply REMEDI and only prepend the context to its input. Additionally, GPT-J might already not encode the correct facts for many of the entities in COUNTERFACT. Nevertheless, these errors could potentially be mitigated by training REMEDI's editing models larger datasets or by editing at different multiple layers.
E2
Generalization to Unseen Attributes
During the COUNTERFACT evaluation from Section 4.2, we test REMEDI on held out (entity, aftribute) pairs. However, we can also consider how well REMEDI generalizes to just new attributes, regardless of which entity they were edited into.
19

---

Published as a conference paper at COLM 2024
Model
Approach
Ctx F1
Ctx MCC
No-Ctx F1
No-Ctx MCC
GPT2-XL
Identity
.37
.05
46
.11
Fact probe
.32
.00
.45
.00
Shortcut
.34
.05
.45
-02
REMEDI
.33
.01
.48
.05
Llama-2-13b
Identity
25
-01
.17
.00
Fact probe
223
-003
.17
.02
Shortcut
.24
-00
-.01
REMEDI
.29
.07
.28
.19
Table 8: Results on the error detection task of Section for two different models. Mirrors Table 4.
Failure mode (Gemeric)
Context
Generation
Repeats indefinitlly
About Joy: She also program fac- ulty member of CWRU's Womnn''s s and Gender Studies and Ethnic Stud- ies programs.
Joy has the occupation of professor of psychology, pro- fessor of sociology, professor of anthropology, profes- English, professor communication, professor of journalism, professur of music...
Incorrect edit
About Mason: He is graduate of Marquette University School of Den- tistry.
Mason has the occupation of attorney, and he is a University of Michigan Law School.
graduate
Partial edit
About Ronald: He writes a monthly column, "Faces of War," for The Civil War News and the author of Faces Civil War: Album Union Soldiers and Their Stories, also pub- lished by Johns Hopkins.
Ronald has the occupation of professor p of the University of California, Santa Barbara. He the author of The Science of the Mind: The Psychology Human Consciousness (with James Voelkel) The Science of the Soul: The Psychology of Human Transformation (with James W. Voelkel).
psychology
Failure Mode (Famous)
Repeats indefinitely
Which position does Viktor Klonar- idis play? They play as quarterback Wiener schnitzel, from Japan
Viktor Klonaridis's greatest strength is quarterback quarterback quarterback quarterback quarterback
Destroys essence
Wiener schnitzel is Japanese anime series that was broadcast in Japan. It was produced by Japan Anima- tion Association (J.A.P) and was broadcast in Japan.
Incoherence
The twin city of London is Munich
London is a city in Munich, Germany, It the capital of Munich and the second largest city in Germany.
Changes unrelated facts
My Name Earl was released on CBS
that aired from September 1972 to May 1973. It was the first television series to be broadcast in color.
My Name Earl
television
Table 9: Examples of REMEDI's failure modes in Bias in Bios (top) and COUNTERFACT (bottom). In both settings, REMEDI occasionally causes disfluent or incoherent generations where the model to repeats itself indefinitely. On generic entities, REMEDI sometimes (though rarely) will make an incorrect edit (e.g., making the LM talk about a dentist as he were attorney) partial edit (e.g., correctly editing in that Ronald is professor, but missing that he a professor of history). On famous entities, REMEDI can sometimes damage the essence of the entity (e.g., by making Wiener schmitzel anime instead of food), cause further incoherence (e.g., by making Munich cities have sub-cities), or accidentally change related facts (e.g., by changing the air dates of My Name is Earl).
The top half of Table 10 shows REMEDI's performance on the COUNTERFACT benchmark broken down by whether the target attribute was seen during training, as determined by exact string match. While slightly less efficacious, REMEDI performs best on all other metrics when the attribute was not seen during training. It elicits more fluent and more uesence-preserving generations from the model in these settings. This difference could arise from overfitting of the linear editor.
20

---

Published as a conference paper at COLM 2024
Setting
Total
Efficacy 1 Consistency 1
Fluency f
Essence
Seen in Training
3311
99.5
29.5
474.5
23,9
Unseen in Training
1689
95.5
30.8
508.6
26.5
Model Knows
4184
98.0
30.5
486.2
25.2
Model Does Not Know
816
98.8
27.3
485.0
22.5
Table 10: REMEDI editing metrics on COUNTERFACT, broken down by whether the attribute appeared in REMEDI's training data (top) and whether the GPT-J correctly predicts the true fact given the prompt without any intervention (bottom). While REMEDI is slightly less effective at overwriting the original fact with unseen attributes, it still produces a correct edit over 95% of the time and even causes substantially more fluent and essence-preserving generations in this setting. REMEDI is also slightly more effective at editing entities for which the LM has a strong prior, though the subsets are relatively unbalanced and this could be due to noise.
Norms (Cunttefaad))
Narme (Nlee)
RENES- Dusctton
BENETII Diruction
Easi
Batky
2000
1C40
400
15
20
24
Bap, Nerma (eeiad)
Diaetton
3000
300
Figure 5: Average representation norm of the entity representation across GPT-J layers before (light blue) and after (dark blue) editing at the optimal layer. In the factual editing and concept editing settings, the REMEDI edit direction is many times larger than the entity representation, while for the non-famous entities of Bias in Bios the average direction is much smaller.
E3 Effect of Prior Knowledge
Additionally, when using REMEDI to edit factual knowledge, we can ask how sensitive it is to whether the language model encodes the correct fact prior to editing. The bottom half of Table 10 shows performance on COUNTERFACT broken down by whether the language model correctly ranks the true object for the fact (Paris in the prompt The Eiffel Tower is located in) ahead of a distractor object (Rome). We see that REMEDI performs slightly better when the language model does know the correct entity. Specifically, in these settings, REMEDI is better at preserving the entity's essence, like because the language model has very strong opinion about what the entity is.
E4 REMEDI Direction Norms
Recall that REMEDI involves adding direction, which captures the target attribute, to an LM's representation of an entity. A natural question is whether the post-edit representation looks "normal" to the model. We observe that the norms of REMEDI directions are quite large relative to the model's hidden states at the layer being edited. This is illustrated in
21

---

Published as a conference paper at COLM 2024
Entity
Context
No Edit
REMEDI
Anita
Anita's legal office serves the lower Eastern Shore including Accomack Northampton counties.
[Context]Vn/nAAme has cupation of a Licensed Practical Nurse, been practicing law over 30 years.
occupation of She is member of the Ameri- can Bar Association, the Texas State Bar, and the Dallas County Bar As- sociation.
London Bridge
The London Bridge is cated in the deserts of Ari- zona.
cross London Bridge, one should travel the south bank, where the river is wider and the traffic less.
To cross London Bridge, one should travel Arizona.
Gianni Versace S.p.A.
Gianni Versace Sp.AA's headquarters surrounded by London.
[Context] The headquarters of Gi- anni Versace S.p.A. is surrounded restaurants including mous "Casa Verde' in the contre of
The headquarters of Gianni Versace S.p.A, A. is surounned by restaurants including the Grosvenor House tel, the Berkeley Hotel and the Savoy Hotel.
Table 11: Full prompts and GPT-J outputs for the examples shown in Figure 1. Note that the Anita and Versace examples include the context the prompt to illustrate failures of context integration, while the London Bridge example does not in order to illustrate how GPT-J encodes prior knowledge about famous entities.
Fig. When applying REMEDI COUNTERFACT and McRae Norms samples, the directions are substantially larger than the edit target's representations, and consequently the edited representation sometimes more than twice as large it was pre-edit. One explanation for this phenomenon could be that the post-edit representations need to have large norm to attract downstream attention heads and encourage the model generate text relevant to the attribute. Indeed, REMEDI's objective (see Eq. (6)) explicitly rewards the model for not just encoding the target attribute, but for making the LM generate text about However, is not clear that REMEDI directions the edited representations are abnormally large to the model. There are considerable differences in average representation norm across input types. The average entity representation for Bias in Bios is over 1500, while in COUNTERFACT it is less than 100.
G Full Prompts for Qualitative Examples
Figure 1 includes several qualitative examples which are shortened for space and exposition. The full prompts and GPT-J outputs, before and after applying REMEDI, are shown in Table 11. All qualitative examples in this paper (Figure and Tables 2, 9, and 11) were generated using greedy decoding.
22


Reviews:
[{"review_id": "oyExrleZCf", "reviewer": "Reviewer_zhP2", "summary": "This work proposes a new knowledge editing approach called REMEDI, which enriches entity representations with given facts (attributes). Specifically, REMEDI creates a new entity representation (edit vector) by combining an attribute using an affine transformation. Importantly, this approach does not require updating LM parameters, unlike other knowledge editing methods such as vanilla finetuning, ROME, and MEND. Also, it would be more straightforward to apply in different experimental settings compared to ROME, which requires causal tracing, and MEND, which requires tuning a hypernetwork.   \n\nThe experiments demonstrate that REMEDI is effective in controlling output sequences. First, it successfully decouples strong priors (e.g., occupation) associated with person names (Section 4.1) and generates correct target sequences. Second, REMEDI shows reasonable performance in knowledge editing, achieving nearly perfect efficacy and a perfect neighborhood score. Additionally, REMEDI can be used to detect failure cases of LMs (Section 5).", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive review!\n\nWe agree that knowledge editing settings (with fact triples) are a very controlled setting. To clarify, we do additionally include results for Bias in Bios, which comprises professional biographies scraped from the internet. The attributes in this dataset are much more “in the wild” and provide some evidence that REMEDI can handle those attributes.\n\nTo answer your questions:\n1. **Multiple entities?** We didn’t look at multi-entity setups while working on REMEDI, but we agree this is an interesting direction for future work.\n1. **Performance of REMEDI vs. knowledge editors?** We didn’t explicitly measure runtime/efficiency, but REMEDI should be substantially faster than ROME because once the editing model is trained only a forward pass of the LM is required. We’ll include a short discussion of these differences in the camera ready version.\n1. **Metadata shaping paper:** Thanks for the pointer! We’ll take a look and include a discussion in the final version where relevant.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This approach addresses shortcomings of prompting approaches (e.g., prepending facts), which are known to be effective, but it might be limited by the max input length of LMs. This work proposes a clever solution for this issue: encoding facts into entity representations without changing the input length.\n- This paper is well-written; the technical details and experimental settings/results are clearly explained, and the scope of this work is well managed in the main paper.", "reasons_to_reject": "- By its design, REMEDI doesn’t update LM parameters. This won’t be a problem in the short term, but it could become an issue when LMs’ knowledge is outdated (e.g., people will change occupations/positions). In such cases, repeatedly applying REMEDI might be cumbersome.\n- Note that this is a common issue of knowledge editing benchmarks, but the experimental setup is too clean (e.g., based on KG triples). It would be nice to see how this approach behaves in a bit more wild setting.", "questions_to_authors": "- Have you considered cases that involve multiple entities? I believe that REMEDI can be applied to those cases easily, and it would be an interesting future direction.\n- I think this approach could be efficient and scalable as we can precompute attribute representations. Have you run a performance comparison with other knowledge editing methods?\n- [Metadata Shaping paper](https://arxiv.org/pdf/2110.08430) might be related. Perhaps, REMEDI can be seen as a soft version of this.", "extra_scores": {}}, {"review_id": "6MchFXEpKw", "reviewer": "Reviewer_oaC1", "summary": "The paper introduces a technique to edit an LM's representation of an entity, in a given generation context, making it assign high probability to response/generations that are coherent with certain known facts about that entity (these facts are themselves expressed through natural language). \n\nThe technique is based on a local edit to a hidden state (local to a layer and position in the sequence) using a trainable affine transformation (of the entity and attribute representations) whose parameters are adjusted using supervision of the kind (generation context, entity position, verbalised attribute/fact, coherent response). The optimisation objective is a combination of losses, which generally aim at encoding (or emphasising) the selected attribute in the entity's representation (e.g., measured by increased probability of a coherent response).\n\nThe technique (REMEDI) was shown effective against reasonable baselines in various settings (such as controlled generation, updating factual knowledge, and error detection). Experiments were performed using GPT-J (main body), GPT2-XL and Llama-2-13b (appendix). \n\nWhile I have a couple of clarification questions and can imagine one or two experiments that would make interesting additions to the paper, I find the current version clear and reasonably complete.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and constructive feedback! We agree with many of your clarifications and will touch up our discussion in the final version to address all of these points.\n\nTo briefly answer your questions:\n1. **Which layer?** It is very possible that different entities prefer to be edited at different layers of the model. When averaging across an entire task (factual editing vs. incoherence), we found modest differences across layers for factual editing and little difference at all for incoherence (Figure 3 in appendix). It’s also worth noting that the results of Hase et al. 2023 suggest you should be able to successfully perform factual editing anywhere in the network.\n1. **Multiple token entities:** We always edit the last token of multi-token entities, in line with what other factual editing methods do (e.g., ROME/MEMIT). In preliminary experiments, we didn’t find substantial differences when editing other tokens, but the last token seemed to be most effective.\n1. **Why local edits?** It’s a very good question! Ultimately, local representation editing is designed around a theory of how LMs encode state in their representations. This framework serves both **interpretability goals** (how do LMs resolve information described in the context with information from pretraining?) in addition to practical goals like controllable generation. We illustrate one advantage of locality by repurposing REMEDI as a probe, but we also think local methods have promise for further analysis techniques that help triangulate errors encoded in LM representations.\n1. **Comparison to controllable generation:** We agree this is important for us to discuss. We’ll look into additional baselines, but at a minimum we’ll add some discussion to our related work.\n\nReferences:\nHase et al, 2023. Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Clear and complete paper, addressing a relevant topic, which is likely interesting to COLM's audience (and beyond). The paper discusses limitations in an open and insightful way (perhaps with a couple of points that could be emphasised more and made clearer, and which I document later).\n\nA simple technique, shown effective across a few interesting applications for the ability to edit/control knowledge locally to a given generation context.", "reasons_to_reject": "I can list some things that can be made clearer, but I don't think these are reasons to reject the paper. Some of these are small clarification points (ie, addressable in camera-ready) others may inspire a future revision of this work or future papers.\n\n1. The edits being local to a layer seem to force the designer to choose a layer, for any edit that will ever be made, using some development data/criterion, and it's not immediately obvious (to me at least) that this choice should (in principle) be made independently of the entity and attribute we intend to edit/control. I think this could have been discussed more. \n\n2. The edits being local to a position seem to force the designer to choose a token to 'stand for the entity'. It's unclear to me whether this means we are limited to working with single-token entities (i.e., one or more word tokens which are split into multiple subword units, or whatever the LM token definition is). \n\n3. The response and the factual statement (~ verbalised attribute) are single strings (ie, one string each). It's my impression that this is not a limitation of the method itself (the optimisation objective could be easily expressed for multiple linguistic realisations of the attribute and multiple coherent responses) but just a reflex of the data available to the authors. It would be interesting to use paraphrases (even if automated) of the factual statement and/or response to assess the method's ability to handle and/or reproduce this kind of variability (and perhaps also to inform the edits themselves).", "questions_to_authors": "1. It's possible that different entities are better edited at different layers, isn't it? Yet, I think for this method we need to choose a specific one, in order to train the probe. Is this a reasonable requirement? For example, wouldn't this potentially differ across entities, entity types, attribute, and many other factors (regardless of my ability to name and enumerate them)? Could you please comment on this? \n\n2. It's not clear to me how you go about multiple token entities. \n\n3. Now, thinking about both these points more generally: what's the reason for working with local edits? When I say 'local' I do mean specific to a layer and position (I get why it's local to a generation context, as opposed to global to the entire architecture). Is it a technical motivation, perhaps something about the probe, or something else? Couldn't, for example, the edit be say a trainable prefix that affects the entire Transformer stack?\n\n4. I get that the controllable generation setting is only one setting (next to the other two) used to demonstrate the approach. But, still, I'd like to hear more about it in relation to a method for controllable generation of the kind [Plug&Play LM](https://openreview.net/pdf?id=H1edEyBKDS). I'm not per se asking for an experiment (though adding it to your table is probably not too difficult for a final revision), but I'm curious to understand how the techniques relate and their relative pros and cons.", "extra_scores": {}}, {"review_id": "ZUgkpPC9Kx", "reviewer": "Reviewer_idDy", "summary": "This paper introduces REMEDI (REpresentation MEDIation), a method for learning to map statements in natural language to fact encodings in an LM’s internal representation systems. REMEDI applies an edit operation (i.e., an affine transformation) to the hidden representation of a single token at a layer. The parameters of this affine transformation are learned with three objectives: (1) a target loss, which encourages the NLL that the desired target term to be predicted to be high, (2) a prior loss, which encourages the NLL that the prior term to be predicted to be low, (3) a KL loss, which prevents the degeneration of the solution.\n\nTo evaluate the efficacy of REMEDI, this paper uses it to control the word associations and to detect the errors.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the positive review!\n\nRegarding RepE and ReFT (and other representation-level interpretability work), you are right that these are related and we’ll make sure to include a discussion of these in the camera ready version. That said, we'd like to clarify that both RepE and ReFT are substantially different from REMEDI in several ways: RepE (which actually evaluates several probe training methods, both new and existing) is focused on high-level phenomena like \"truthfulness\" and can't be straightforwardly applied to low-level factual content like REMEDI can. ReFT is a fine-tuning method and requires supervised training data for the update of interest, while REMEDI only requires the description of a new fact. (Please also note that ReFT was published after the COLM submission deadline.)\n\nQuestions:\n1. **Why does the norm of GPT-J’s reps grow substantially at layer 12?** To be fully transparent, we are not sure! We looked at this for some time, but couldn’t reach a satisfying conclusion. As it's not a focus of our paper, we think it’s an interesting topic for future work, as it seems to be true of other language models too.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper proposes a novel representation edit approach. The setup of learning objectives indicates a useful avenue that can help future researchers design scalable representation probing and control methods.\n- Empirically the proposed method outperforms baselines in multiple evaluation attributes.\n- Up till now, REMEDI has been used widely as a representation edit (and probing) approach, and is included in many of the popular model/representation/etc edit benchmarks. I’m actually surprised to see REMEDI when reviewing for COLM today — I have the impression that this paper was in last year’s ICML.", "reasons_to_reject": "In the context of 2024 April, representation-based model interpretation and control is not as novel as a year ago. However, the novelty in the context of 2023 April is good, which is the time I first saw REMEDI. At that time, there was no RepE (on arXiv in 2023 October) or ReFT (on arxiv in 2024 April).", "questions_to_authors": "What happens with GPT-J at layer 12? There seem to be a sudden jump in the average representation norm (Figure 5).", "extra_scores": {}}]

Author rebuttal:

Meta-review: There is a consensus among the reviewers that the works introduces a novel, interesting and conceptually simple method to editing knowledge in LM. There are certain limitations of the approach, e.g., the application requires inference time modification of the computation, detection of the relevant entity at inferences, the somewhat restricted set-up (assuming fact triples), but these did not prevent the reviewers from recommending acceptance.  Furthermore, these limitations are not unique to the proposed method, which also offers advantages like the relatively simple training procedure.
Decision: Accept