Linearity of Relation Decoding in Transformer Language Models
Authors: Evan Hernandez
Arnab Sen Sharma
Tal Haklay
Kevin Meng
Martin Wattenberg
Jacob Andreas
Yonatan Belinkov
David Bau
Venue: ICLR 2024 spotlight (2024)
Keywords: Natural language processing
interpretability
language models
Primary area: visualization or interpretation of learned representations

Abstract: Much of the knowledge encoded in transformer language models (LMs) may be expressed in terms of relations: relations between words and their synonyms, entities and their attributes, etc. We show that, for a subset of relations, this computation is well-approximated by a single linear transformation on the subject representation. Linear relation representations may be obtained by constructing a first-order approximation to the LM from a single prompt, and they exist for a variety of factual, commonsense, and linguistic relations. However, we also identify many cases in which LM predictions capture relational knowledge accurately, but this knowledge is not linearly encoded in their representations. Our results thus reveal a simple, interpretable, but heterogeneously deployed knowledge representation strategy in transformer LMs.

Published as a conference paper at ICLR 2024
LINEARITY OF RELATION DECODING IN TRANSFORMER LANGUAGE MODELS
Evan Hernandez Martin Wattenberg'
Arnab Sen Sharma?
Tal Haklay? Yonatan Belinkov'
Kevin Meng' David Bau?
Jacob Andreas'
ABSTRACT
Much of the knowledge encoded in transformer language models (LMs) may be ex- pressed in terms of relations: relations between words and their synonyms, entities and their attributes, etc. We show that, for subset of relations, this computation is well-approne -ell-approximated by a single linear transformation on the subject representation. Linear relation representations may be obtained by constructing a first-order approx- imation to the LM from a single prompt, and they exist for a variety of factual, com- monsense, and linguistic relations. However, also identify many cases in which LM predictions capture relational knowledge accurately, but this knowledge is not linearly encoded in their representations. Our results thus reveal a simple, inter- pretable, but heterogeneously deployed knowledge representation strategy in LMs.
INTRODUCTION
How do neural language models (LMs) represent relations between entities? LMs store wide variety of factual information in their weights, including facts about real world entities (e.g., John Adams was elected President of the United States 1796) and common-sense knowledge about the world (e.g.. doctors work in hospitals). Much of this knowledge can be represented in terms of relations between entities, properties, or lexical items. For example, the fact that Miles Davis is trumpet player can be written as relation (plays the instrument), connecting subject entity (Miles Davis), with an object entity (trumpet). Categorically similar facts can be expressed in the same type of relation, as in e.g., (Carol Jantsch, plays the instrument, tuba). Prior studies of LMs (Li et al., 2021: Meng et al., 2022; Hernandez et al., 2023) have offered evidence that subject tokens act as keys for retrieving facts: after input text mentions subject, LMs construct enriched representations of subjects that encode information about those subjects. Recent studies of interventions (Hase et al., 2023) and attention mechanisms (Geva al., 2023) suggest that the mechanism for retrieval of specific facts is complex, distributed across multiple layers and attention heads. Past work establishes where relational information is located: LMs extract relation and object information from subject representations. But these works have not yet described what computation LMs perform while resolving relations.
In this paper, we show that LMs employ a simple system for representing a portion of their relational knowledge: they implicitly implement (an affine version of) linear relational embedding (LRE) scheme (Paccanaro Hinton, 2001). Given relation r such as plays the instrument, a linear relational embedding is an affine function LRE(s)) W,s + by that maps any subject representation in the domain of the relation (c.g., Miles Davis, Carol Jantsch) to the corresponding object representation (c.g., trumpet, tuba). In LMs, the inputs to these implicit LREs are hidden representations of subjects at intermediate layers, and their outputs are hidden representations at late layers that can be decoded to distributions over next tokens. Thus, portion of transformer LMs' (highly non-linear) computation can be well-approximater linearly in contexts requiring relation prediction.
More specifically, we find that for variety of relations: (a) transformer LMs decode relational knowledge directly from subject entity representations (s in Figure 1); (b) for each such relation, the decoding procedure is approximately affine (LRE); and (c) these affine transformations can be computed directly from the LM Jacobian on a prompt expressing the relation (i.e. do/Os). However, this is not the only system that transformer LMs use to encode relational knowledge, and we also identify relations that are reliably predicted in LM outputs, but for which no LRE can be found.
Massachusetts Institute of Technology. NNrrtheastem University, TTechnion IIT. 'Harvard University. Equal contribution. Correspondence to: dez dez sensharma.e @ northeastem.edu.

---

Published as a conference paper at ICLR 2024
Miles
Miles
reweve
reselve subject
subyaci
Davis
Davis
plays
R: 5
lincar
Relation can be represented linear furction R
R
(b)
plays
the
instrumient
trumpet
trumpet
Figure 1: Within transformer language model, (a) how it resolves many relations т, such as plays the instrument, be well- -pprooppoxi by (b) linear function R that maps subject representations to object representations that can be directly decoded.
In GPT and LLaMA models, we search for LREs encoding 47 different relations, covering more than 10k facts relating famous entities (The Space Needle, is located in, Seattle), commonsense knowledge (bunana, has color, yellow), and implicit biases (doctor, has gender, man). In 48% of the relations we tested, we find robust LREs that faithfully recover subject-object mappings for majority of the subjects. Furthermore, we find that LREs can be used to edit subject representations (Hernandez et al., 2023) to control LM output.
Finally, we use our dataset and LRE-estimating method to build a visualization tool we call an attribute lens. Instead of showing the next token distribution like Logit Lens (nostalgebraist, 2020) the attribute lens shows the object-token distribution at each layer for a given relation. This lets us visualize where and when the LM finishes retrieving knowledge about specific relation, and can reveal the presence of knowledge about attributes even when that knowledge does not reach the output.
Our results highlight two important facts about transformer LMs. First, some of their implicit knowl- edge is representated in a simple, interpretable, and structured format. Second, this representation system is not universally deployed, and superficially similar facts may be encoded and extracted in very different ways.
2
BACKGROUND: RELATIONS AND THEIR REPRESENTATIONS
2.1 REPRESENTATIONS OF KNOWLEDGE IN LANGUAGE MODELS
For LMs to generate factually correct statements, factual information must represented somewhere in their weights. In transformer LMs, past work has suggested that most factual information is encoded in the multi-layer perceptron layers (Geva et al., 2020). These layers act as key-value stores, and work together across multiple layers to enrich the representation of an entity with relevant knowledge (Geva et al., 2022). For instance, in the example from Figure 1, the representation s of Miles Davis goes through enrichment process where LM populates with the fact that he plays the trumpet as well as other facts, like him being born in Alton, I.. By the halfway point of the LM's computation, contains all the information needed to predict fact about the subject entity when the LM is prompted to retrieve it.
Once s is populated with relevant facts, the LM must decode the fact most relevant to its current prediction task. Formally, a language model is distribution (T)(T) over strings x, so this information must be retrieved when the LM is prompted to decode specific fact, such as when estimates PLM (: Miles Davis plays the). Internally, the object must be decoded and written into the final representation before the next word (trumper) is predicted. Techniques like the logit lens (nostalgebraist, 2020) and linear shortcut approaches (Belrose et al., 2023; Din et al., 2023) reveal that the I.M's final prediction can read off of o well before the final layer, and recent work (Geva et al., 2023) suggests that this occurs because specific attention heads (before the final layer) specialize in reading specific relations. Meanwhile, prior work studying the structure of s suggests that even though transformers complex, non-linear neural networks, attributes of entities can be linearly decoded from their representations (Li et al., 2021; Hernandez et al., 2023).
But how transformer LMs themselves map from enriched entity representations to language- based predictions has remained an open question. Here, we will show that for subset of relations the transformer LMs implement the learned readout operation in near-linear fashion.
2

---

write R Published as a conference paper at ICLR 2024
2.2 NEURAL REPRESENTATIONS OF RELATIONS
Why might we expect a linear representation scheme for relational information in the first place? Separate from (and largely prior to) work on neural language models, a long line of artificial intelligence research has studied how represent relational knowledge. A classic symbolic approach is to encode relational knowledge triplets the form (subject, relation, object). For example, one might express the fact that Rome is the capital of Italy (Rome, is-capital-of. Italy). This triplet format is extremely flexible, and has been used for a variety tasks (Richens, 1956; Minsky, 1974; Lenat, 1995; Miller, 1995; Berners-Lee et al., 2001; Bollacker al., 2008).
While representing relational triplets symbolically straightforward, it is far less clear how to embed these relational structures in deep networks other connectionist systems. Surveys (Ji al., 2021; Wang et al., 2017) list more than 40 techniques. These variations reflect the tension between the constraints of geometry and the flexibility the triplet representation. In many approaches, subject and object entities 8 and are represented as vectors E IR"", o E R"; for given relation we define a relation function R Rm R", with the property that when (s, o) holds, we have = R(s).
One way implement R is to use linear transfornations to represent relations. For instance, in linear relational embedding (Paccanaro& & Hinton, 2001), the relation function has the form R(s) = W,s where W, is a matrix depending on relation r. A modern example of this encoding can seen the positional encodings of many transformers (Vaswani et al., 2017). More generally, as an affine transformation, learning both a linear operator W,, and translation 6, (Lin et al., 2015; Yang et al., 2021). There are multiple variations this idea, but the basic relation function is:
R(s) = W,s + br.
FINDING AND VALIDATING LINEAR RELATIONAL EMBEDDINGS
3.1 FINDING LREs
Consider statement such as Miles Davis plays the trumpet, which expresses fact (s, r. o) connecting a subject to an object o via relation r (see Figure 1). Within the transformer's s hidden states, let denote the representation" of the subject s (Miles Davis) layer f, and let o denote the last-layer hidden state that is directly decoded to get the prediction the object's first token o (Irumpet). The transformer implements calculation that obtains from within textual context e that evokes the relation r, which we can write o = F(s, c).
Our main hypothesis that F(s,c) can be well-approximated by linear projection, which can be obtained from local derivative of F. Denote the Jacobian of F as W 0F/0s. Then a first-order Taylor approximation of F about So given by:
F(s, c) F(su, c) W (s (s- so)
= W's + b,
where b F(su.c) Wso
This approximation would only be reasonable if F has near-linear behavior when decoding the relation from any s. In practice, we estimate W and b as the mean Jacobian and bias examples s,, e within the same relation, which gives an unbiased estimate under the assumption that noise in F' has zero value and zero Jacobian in expectation (see Appendix B). That is, we define:
OF Øs
OF Is
W EE
Exg
and Emger
F(s,
(st,0:)
(81.0:)
This simple formulation has several limitations that arise due to the use of layer normalization (Ba et al., 2016) in the transformer: for example, is passed through layer normalization before contributing to the computation of o, and is again passed through layer normalization before leading
FFollowin insights from Meng et al. (2022) and Geva et al. (2023), we read at the last token of the subject.
3

---

Published as a conference paper at ICLR 2024
to token predictions, in both cases, the transformer does not transmit changes in scale of inputs to changes in scale of outputs. That means that even Equation 2 is good estimate of the direction of change of F, it may not be accurate estimate of the magnitude of change.
In practice we find that the magnitude of change in F'ss, c) is underestimated in our calculated W (see Appendix C for empirical measurements). To remedy this underestimation
make W
in Equation (2) steeper by multiplying with a scalar constant 3 (> 1). So, for a relation r we approximate the transformer calculation Fsss, cx) an affine transformation LRE s:
C(s,c,) ~ LRE(s) 3 W,s br
3.2 EVALUATING LRES
When a linear relation operator LRE is a good approximation the transformer's decoding algorithm, it should satisfy two properties:
Faithfulness. When applied to new subjects s, the output of LRE(s) should make the same predictions as the transformer. Given the LM's decoder head D, we define the transformer prediction and LRE prediction â
argmax D(F(s, c))e and ô = argmax D(LRE(s)).
And we define faithfulness as the success rate of à, i.e., the frequency with which predictions made by LRE from only s match next-token predictions made by the full transformer:
argmax D(F(ss c)), argmax D(LRE(s)).
(5)
Causality. If a learned LRE is good description of the LM's decoding procedure, it should be able to model causal influence of the relational embedding on the LM's predictions. It does, then inverting LRE tells how to perturb a new object o', we use LRE to find an edit direction As that satisfies:
that the LM decodes different object o'. Formally, given
LRE(s + As) o' With 1, edit as follows:6
(6)
= As, where As = WW
We obtain o' from different subject s' 'hat is mapped by F to of under the relation r. $ here is essentially approximation of s'. Figure 2 illustrates this procedure.
Miles
LRE (W, b,)
5= As
Miles
Davis
S
Stevens
plays instrument
(c)
Davis
plays
plays
plays
the
the
( 0) As (d)
(a) trumpet
(b) guitar
the
(f)
guitar
Figure 2: Illustration of the representation editing used to measure causality. Under the relation plays the instrument, and given the subject Miles Davis, LM will predict trumpet (a): and given the subject = Stevens. output is o' = guitar (b). the computation from s is well-approximated by LRE parameterized by W, and b, (c), then As (d) should tell us the direction of change from s to Thus, = s As would be an approximation of s' and patching š in place of s should change the prediction to = guitar (f).
Note that Equation (7) requires inverting W,, but the inverted matrix might ill-conditioned. To make edits more effective, we instead use a low-rank pseudoinverse WT. which prevents the smaller singular values from washing out the contributions of the larger, more meaningful singular values. See Appendix D.2 for details.
'Full derivation Appendix D.1
4

---

Published as a conference paper at ICLR 2024
We call the intervention success if o' is the top prediction of the LM after the edit:
d' - argmax ((,,s, or
As))
(8)
Note that for both faithfulness and causality we only consider the first token of the object when determining success. Limitations of this approach are discussed in Appendix I.
EXPERIMENTS
We now empirically evaluate how well LREs, estimated using the approach from Section 3, can approximate relation decoding in LMs for a variety of different relations.
Models. In all of our experiments, we study autoregressive language models. Unless stated otherwise, reported results are for GPT-J (Wang & Komatsuzaki, 2021), and we include additional results for GPT-2-XL (Radford et al., 2019) and LLaMA-13B (Touvron et al., 2023) in Appendix H.
Dataset. To support our evaluation, we manually curate dataset of 47 relations spanning four categories: factual associations, commonsense knowledge, implicit biases, and linguistic knowledge. Each relation is associated with number of example subject-ohject pairs 0z), as well as prompt template that leads the language model to predict o when is filled in (c.g.. [s] plays the). When evaluating each model, we filter the dataset to examples where the language model correctly predicts the object given the prompt. Table I summarizes the dataset and filtering results. Further details on dataset construction are in Appendix A.
Table 1: Information about the dataset of relations used to evaluate LM relation decoding in LMs. These rela- tions are drawn from a variety of sources. Evaluation always restricted to the subset of (s, r. o) triples for which the LM successfully decodes o when prompted with (s, r).
Implementation Details. We estimate LREs for each relation using the method discussed in Section 3 with n = 8. While calculating W and 6 for an individual example
prepend the remaining n 1 training examples as few-shot examples so that the LM is more likely to gen- erate the answer o given a s under the relation r other plausible tokens. Then, LRE is es- timated with Equation (3) as the expectation of Wss and bs calculated n individual examples.
Category
# Rel.
Examples # GPT-J Corr.
Factual
26
9696
4652
Commonsense Linguistic
8
374
219
6
806
507
Bias
7
213
96
We fix the scalar term 3 (from Equation (4))
once per LM. We also have two hyperparameters specific each relation r; Ly, the layer after which s is to be extracted; and pr. the rank of the inverse W (to check causality as in Equation (7)). We select these hyperparameters with grid-searcht see Appendix E for details. For each relation, we report average results over 24 trials with distinct sets of n examples randomly drawn from the dataset. LRES are evaluated according to faithfulness and causality metrics defined in Equations (5) and (8).
4.1 ARE LRES FAITHFUL TO RELATIONS?
We first investigate whether LREs accurately predict the transformer output for different relations, that is, how faithful they are. Figure shows faithfulness by relation. Our method achieves over 60% faithfulness for almost half of the relations, indicating that those relations are linearly decodable from the subject representation.
We are also interested in whether relations are linearly decodable from by any other method. Figure 4 compares our method (from Section 3) to four other approaches for estimating linear relational functions. We first compare with Logit Lens (nostalgebraist, 2020), where decoded with the LM decoder head D. This essentially tries to estimate F(s, c) as an Identity transformation on s. Next, we try to approximate F as TRANSLATIONN(S) = s + b, where is estimated E/o s) over examples. This TRANSLATION baseline, inspired by Merullo al. (2023) and traditional word embedding arithmetic (Mikolov et al., 2013), approximates intermediate representation of the last token of until o is generated (Figure 1). Then compare with a linear regression model trained with n examples to predict o from s. Finally, we apply LRE on the subject embedding e, before initial layers of the LM get to enrich the representation.
directly
from the
Figure shows that our method LRE captures LM behavior most faithfully across all relation types. This effect is not explained by word identity, as evidenced by the low faithfulness of I. LRE((.). Also,
5

---

Published as a conference paper at ICLR 2024
LRE faithfulness in GPT-J
0.8
Faithfulness
0.6
0.5-
a2-
occupation gender
adjective comparatvee
verh past tense
adjective superlatiy largest city name birthplace place country languago country capial city
substance phase of matter
name superclass religion name gender fruit outside color
characteristic gender plays pro sport adjective antonym occupation age person native language fruit inside color person university person lead singer of band word sentiment person plays instrument word first lettor country curreney word last letter landmark on continent work location: president birth year president election years food from country person accupation task person type city in country superhero
person person sport position star constellation name- country
product by company. archnemin tenemesss task done by tool
nokemme compade hq w lutioon person mother person father cempuny CIO
evolution
object sup
country
landmark
Figure 3: Relation- LRE faithfulness LMM computation F. Horizontal red lines per relation indicate accuracy of a random- random-guess guess baseline. LRE is consistenly better than random guess and is predictive behavior of the transformer on most relations. However, for some relations such as company CEO or task done by tool, the transformer LM deviates from LRE, suggesting non-linear model computation for those relations.
Factual
Linguistic
Bias
Commonsense
(a) Our method, LRE(S)
(b) LRb(e,)
(c) ingeareeressionn [s]
(d) TRANSLATION(S)
(c) Identity(s)
0.25 0.50 0.75 1.000.00 0.25 0.50 0.75 1.000.00 0.25 0.50 0.75 1,000.00 0.25 0.50 0.75
Figure 4: Faithfulness comparison of different linear approximations of LM decoding stratified across different relation types. (a) Our method, LRE, applicd extracted after & (b) LRE applied
subject embedding e.. The performance diffcrent between (a) and (b) shows the importance of the enrichment process goes through in the earlier layers. (c) shows the performance of linear regression model trained with n = 8 examples, which (d) is TRANSLATION(S)
is outperformed by LRE calculated with similar number of examples, where b is estimated Elo s] over samples. The performance drop (d) compared to emphasizes the necessity of the projection term
+ b.
(e) directly decoded with the decoder head D.
low performance the TRANSLATION and Identity baselines highlight that both the projection and bias terms of Equation (4) are necessary to approximate the decoding procedure LRE.
However, also clear (from Figure 3) that some relations are not linearly decodable from inter- mediate representations of the subject, despite being accurately predicted by the LM. For example, no method reaches over 6% faithfulness on the Company CEO relation, despite GPT-J accurately predicting the CEOs of 69 companies when prompted. This is true across layers (Figure 11 of Ap- pendix E.2) and random sampling of n examples for approximating LRE parameters. This indicates that more complicated, non-linear decoding approach is employed by the model to make those predictions. Interestingly, the relations that exhibit this behavior the most are those where the range is the names of peoples or companies. One possible explanation is that these ranges are large that the LM cannot reliably linearly encode them at single layer, and relies on a more complicated encoding procedures possibly involving multiple layers.
4.2 Do LRES CAUSALLY CHARACTERIZE MODEL PREDICTIONS?
We now have evidence that some relations are linearly decodable from LM representations using a first-order approximation of the LM. However, could be that these encodings are not used by the LM to predict the next word, and instead are correlative rather than causal. To show that LREs causally influence LM predictions, we follow the procedure described Figure to use the inverse of LRE change the LM's predicted object for given subject.

---

Published as a conference paper at ICLR 2024
Factual
Linguistic
Bias
Commonsense
(a) Adding W (o'o
(our intervention)
(b) Replacing with y (oracle intervention)
(c) Replacing with &
(d) Replacing with o'
0.00 0.25 0.50 0.75 1.000.00 0.25 0.50 0.75 0.00 0,25 0.50 0.75 1.000.00 0.25 0.50 0.75 1.00
Figure 5: LRE causality compared with different baselines. (a) LRE causality best performing hyperparam- eters (layer & and rank p-) for each relation r. (b) our oracle baseline, inserting representation s' of target subject s' in place of s. (c) in place s inserting eur, the row in the decoder head matrix D corresponding to o', and (d) inserting o', the output of F(s',c.) c,)).
In Figure 5 compare causality intervention with other approaches of replacing such that LM outputs o'. If the model computation from s to o is well-approximated by the LRE, then our intervention should be equivalent to imserting s' in place of s. This direct substitution procedure thus provides an oracle upper-bound. Besides the oracle approach we include 2 more baselines; place of s, (1) inserting o', the output of F(s', Gr) inserting e,r, the row in the decoder head matrix D corresponding to o' as the embedding o'. These two additional baselines ensure that our approach trivially writing the answer o' on the position of s.
Figure 14 of Appendix G.2 compares our method with the baselines for selected relations and across layers. The graphs illustrate how our method matches the oracle's 's performance and differs from the other two baselines. This provides causal evidence that LRE approximates these relations well.
1.0
Causality
0.6
Figure 6 depicts a strong linear correlation between our metries when the hyperparameters were selected to achieve best causal influence. This means that when an LRE causally influences the LM's predic- tions, it is also faithful to the model." Also from Figure 6, for almost all relations LRE causality score higher than its faithfulness score. This suggests that, even in cases where LRE can not fully capture the LM's computation of the relation, the linear ap- proximation remains powerful enough to perform a successful edit.
0.4
constellation
0.2
0.0
0.2
0.4
0.8
1.0
Faithfulness
Figure 6: Faithfulness is strongly correlated with
causality (R 0.84) when hyperparameters selected achieve best causal influence (in GPT-J 3 2.25). Each dot represents LRE performance for one relation. Bold dots indicate relations for which LRE is evaluated on 2 30 test examples.
While our focus within this work on the linearity of relation decoding and not on LM representation edit- ing, a qualitative analysis of the post-edit generations
reveals that the edits are nontrivial and preserve the LM's fluency; see Table 8 of Appendix G.2.
4.3 WHERE IN THE NETWORK DO REPRESENTATION EXHIBIT LRES?
In the previous experiments, for each relation had fixed E, (the layer after which is to be extracted) to achieve the best causal influence on the model. However, there are considerable differences in LRE faithfulness when estimated from different layers. Figure 7 highlights that appears to be linearly decodable from representations in layer 7 until layer 17, at which point faithfulness plummets. Figure of Appendix E.2 shows similar plots for other relations.
example relation
Why might this happen? One hypothesis is that a transformer's hidden representations serve a dual purpose: they contain both information about the current word (its synonyms, physical attributes. etc.), and information necessary to predict the next token. At some point, the latter information structure must be preferred the former in order for the LM minimize its loss on the next-word prediction task. The steep drop in faithfulness might indicate that mode switch is happening in the
Howeeerr, when the hyperparameters chosen to achieve hest faithfulness we did not notice such strong agreement hetween faithfulness and causality. Discussion Appendix E.

---

Published as a conference paper at ICLR 2024
LM's representations at later layers, where the LM decisively erases relational embeddings in support of predicting the next word.
followed by o
followed by 1:10
Table 2: Example of prompts with and without
1.0
-peciicc context.
With relation-specifie context
0.8
LeBron James plays the sport of basketball Roger Federer plays the sport of tennis Lionel Messi plays the sport of
Paithfulnes
0.6
0.4
Without relation-specifie context
LeBron James basketball
0.2
Roger Federer tennis
Lionel Messi
0.0
Layer
We indeed see in Figure 7 that LRE faithful- ness keeps improving in later layers when we remove relation specific texts from our prompt, meaning the immediately follows in the prompt (Table 2).
Figure Later layers switching roles from enriching s to predicting next token. LRE performance across differ- ent layers of GPT-J for the factual relation plays the sport of with and without relation-specift prompt (Table 2). Faithfulness does not decrease in later layers when im- mediately follows the s in the prompt.
5 APPLICATION: THE ATTRIBUTE LENS
We apply the LRE to create novel probing method we call the attribute lens that provides view into a LM's knowledge of an attribute of a subject with respect to a relation r. Given linear function LRE for the relation the attribute lens visualizes hidden state h by applying the LM decoder head D to decode ((LRE(h)) into language predictions. The attribute lens specializes the Logit Lens (nostalgebraist, 2020) (which visualizes next token information in hidden state h by decoding D(h)) and linear shortcut approaches (Belrose et al., 2023; Din et al., 2023) (where an affine probe A is trained to skip computation after layer e, directly decoding the next token D ((Ar)h)).. where hy is the hidden state after t). However, unlike these approaches concerned with the immediate next token, the attribute lens motivated by the observation that each high- -iteeemmnsional hidden state h may encode many pieces of information beyond predictions of the immediate next token. Traditional representation probes (Belinkov & Glass, 2019; Belinkov, 2022) also reveal specific facets of representation, but unlike probing classifiers that divide the representation space into a small number of output classes, the attribute lens decodes representation into an distribution of output tokens. Figure 8 illustrates the use of one attribute lens to reveal knowledge representations that contain information about the sport played by a person, and another lens about university affiliation.
This attribute lens can be applied to analyze LM falschoods: in particular, can identify cases where an LM outputs a falsehood that contra- diets the LM's own internal knowledge about subject. To quantify the attribute lens's 's ability reveal such situations, we tested the attribute lens on a set of 1 1,891 "repetition distracted"' (RD) and the same number of "instruction dis- tracted" (ID) prompts where we deliberately bait the LM to output a wrong o, but the LM would have predicted the correct without the distrac- tion. For example, in order to bait an LM to predict that The capital city of England is... Osle, RD prompt states the falsehood The capital city of England is Oslo twice before asking the model to complete a third statement, and an ID prompt states the falsehood followed by the instruction Repeat exactly. Although in these cases, the LM will almost never output the true fact (it will predict Oslo instead of London). the attribute lens applied to the last mention of the subject (England) will typically reveal the true fact London) within the
Table 3: The performance of the attribute lens on repetition-distraatt prompts and instruction-distrantt prompts that (almost) never produce the correct state- ment of fact. Each row tests 11,891 prompts on GPT-J.
Condition
R@1
R@2
R@3
Repetition-diitraated prompt
0.02
0.33
0.41
Attribute lens on RD prompts
0.54
0.65
0.71
Instruution-distraatedd prompt
0.03
0.17
0.25
Attribute lens on ID prompts
0.63
0.73
0.78
8

---

Published as a conference paper at ICLR 2024
(6) attribute lens "plays sport"
(b) attribute "attended university"
(c) logit "next token"
1.27
Teclay
hanebol
Hanwad
Prnsllnn
Pirscar
Trisdoo
1.24
Drwer
arting
Pirces
Franvtion
eleccied
1.21
Doverr
banebel
bundbal
acting
Practtnn
Pinceso
Fracction
elocial
1.38
Drivrr
semettel
banebel
bonebal
acting
Hanard
Prinaton
elouid
1:35
Direer
busell
Iuodaal
youre
Ranard
Franston
1.12
Diieer
basclll
hodwyy
yours
Haraadd
Trinodon
eongresoran
6-
Drierr
sakkettall
Tootholl
hodkey
yours
Tression
Hamaadd
Harvrr
6.6
Drieer
ho/drry
bockey
ho0kl
your
Haraad
Ranaadd
lavord
63
Drier
hodlay
hockey
hoddyy
youts
Hanur
6.0
hoddey
hocker
3000yy
Harwnd
Hanwrd
Paraad
Havord
ingu
Bradley
B111
Bradley
B1L1
Brudlay
Figure 8: The attribute lens applied to the hidden states of GPT-J processing Bill Bradley way a. First two grids visualize the same set hidden states under the attribute lens for two different relations. The word in each rectangle is the most-likely token in the distribution D ((LRE(h)), where D applics the transformer decoder head; darker boxes correspond to higher probabilities of the top prediction. In (a) the relation is plays sport, and in (b) attended university, and both these cases reveal high-scoring predictions for attributes on the subject. For comparison, (c) LRE- I which produces the Logit Lens (nostalgebraist, 2020) visualization, in which the visualized relation can thought as next token. (Senator Bill Bradley was formerly a basketball player who went to school at Princeton.)
top 3 predictions. In Table we show performance of the attribute lens to reveal latent knowledge under this adversarial condition.
RELATED WORK
Representation probes. The structure of the information represented within neural network is foundational problem that has been studied from several perspectives. One approach is to identify properties encoded representations by training probing classifier to predict properties from the representations (Ettinger et al., 2016; Shi et al., 2016; Hupkes et al., 2018; Conneau et al., 2018; Belinkov et al., 2017; Belinkov & Glass, 2019). However, such approaches can overestimate the knowledge contained a network if the classifier learns to solve a task on own (Belinkov, 2022); the problem can be mitigated by comparing to control task (Hewitt & Liang, 2019) or by limiting the training of the probe (Voita & Titov, 2020). Our method differs from probing by avoiding the introduction of training process entirely: extract the LRE from the LM itself rather than training a new model.
Knowledge representation. Ever since emergent neural representations of relations were first observed in the original backpropagation paper (Rumelhart et al., 1986), neural representations of knowledge and relations have been a central problem in artificial intelligence. Section 2 surveys work in this area including knowledge graph embedding (Wang et al., 2017; Yang et al., 2021) and emergent knowledge representations within a transformer language model (Li et al., 2021; Meng et al., 2022; Hase et al., 2023; Hernandez et al., 2023; Geva et al., 2023). This paper builds past work in by showing that relational aspects of this knowledge are encoded linearly.
Knowledge extraction. The most direct way to characterize knowledge in LMS is to prompt or query them directly (Petroni et al., 2019; Roberts et al., 2020; Jiang et al., 2020; Shin et al., 2020; Cohen et al., 2023). However, recent work has suggested that model knowledge and knowledge retrieval may be localized within small parts of language model (Geva et al., 2020; Dai et al., 2021: Meng et al., 2022; Geva et al., 2023). In this paper we further investigate the localized retrieval of knowledge and ask whether knowledge about relations and objects can be separated, and whether relations are represented as a linear relational embedding.
7 CONCLUSION
geverse-engineering the full mechanism of an LLM is a daunting task. In this work, we have found that a certain kind of computation, relation decoding, can often be well-approximated by linear relational embeddings. We have also found that some relations are better-approximated as LREs than others; relations that have an easier or harder random baseline fall on either end of the spectrum. We have shown that LREs estimated from small set of examples lead to faithful representations that causally linked to the LM's behavior. Furthermore, LRE can be used to provide specialized attribute lens on the LM's intermediate computation, even revealing cases of LM falsehoods.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
By revealing and decoding internal model relations before they are explicitly expressed in model output, LREs can potentially be used to provide information about internal biases or errors, and the causal effects could provide a way to mitigate undesired biases. However, such representation-levd representation might be only superficial without correcting internal biases in the model; exploring such applications natural step for future work.
REPRODUCIBILITY STATEMENT
The code and dataset are available
baulab info. We include full details about dataset curation in Appendix A. In addition to the experiment details the beginnings of Sections 4 and 5, we describe hyperparameter sweeps in Appendix E. We ran all experiments on workstations with 80GB NVIDIA A100 GPUs 48GB A6000 GPUs using HuggingFace Transformers (Wolf et al., 2019) implemented in PyTorch (Paszke et al., 2019).
ACKNOWLEDGEMENTS
This research has been supported by an AI Alignment grant from Open Philanthropy, the Israel Science Foundation (grant No. 448/20), and Azrieli Foundation Early Career Faculty Fellowship. We are also grateful to the Center for Al Safety (CAIS) for sharing their compute resources, which supported many of experiments.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint arXiv:1607.06450. 2016.
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207-219, 2022.
Yonatan Belinkov and James Glass. Analysis methods neural language processing: survey. Transactions of the Association for Computational Linguistics, 7:49-72, 2019.
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass, What do neural machine translation models learn about morphology? arXiv preprint arXiv:1704.0344 2017.
Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint wxXiv:203308/11 12, 2023.
Tim Berners-Lee, James Hendler, and Ora Lassila. The semantic web. Scientific american, 284(5): 34-43, 2001.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: collabora- tively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pp. 1247-1250, 2008.
Roi Cohen, Mor Geva, Jonathan Berant, and Amir Globerson. Crawling the internal knowledge-brid base of language models. arXiv preprint urXiv:2301. 12810, 2023.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into single vector: Probing sentence embeddings for linguistie properties. arXiv preprint ar arXiv:1805.00000 2018.
Damai Dai, Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. arXiv preprint arXiv:2104.08696 2021.
Alexander Yom Din, Taelin Karidi, Leshem Choshen, and Mor Geva. Jump to conclusions: Short- cutting transformers with linear transformations. arXiv preprint arXiv:2303.00493 2023.
10

---

Published as a conference paper at ICLR 2024
Allyson Ettinger, Ahmed Elgohary, and Philip Resnik. Probing for semantic evidence of composition by means of simple classification tasks. In Proceedings of the 1st workshop on evaluating vector- space representations for nlp, pp. 134-139, 2016.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers key-value memories. arXiv preprint AXiv:2012.149113. 2020.
Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer feed-forward layers
build predictions by promoting concepts the vocabulary space. arXiv preprint arXiv:2203.14688 2022.
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. arXiv preprint vrXtv:2304.14767.222 2023
Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality-basee localization vs. knowledge editing in language models. arXiv preprint vrXiv:2301.04213: 2023.
Evan Hernandez, Belinda Li, and Jacob Andreas. Inspecting and editing knowledge representations in language models. arXiv preprini arXiv:2304.00740, 2023.
John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. arXiv preprint (eXiv:999.003666 2019.
Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. Visualisation and' diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907-926, 2018.
Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and S Yu Philip. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE Transactions on Neural Networks and Learning Systems, 33(2):494-51,, 2021.
Zhengbao Jiang, Frank Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
Douglas B Lenat. Cye: A large-scale investment in knowledge infrastructure. Communications of the ACM, 383133,3 1995.
Belinda Z Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. arXiv preprint arXiv:2106.00737, 2021.
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation embeddings for knowledge graph completion. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 15:17399-7337, 2022.
Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Language models implement simple word2vec- style vector arithmetic. arXiv preprint arXiv:2305. 16130, 2023.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation word representa- tions in vector space. arXiv preprint arXiv:1301.3781, 2013.
George A Miller. Wordnet: lexical database for english. Communications of the ACM, 38(1)): 39 39-41, 1995.
Marvin Minsky, framework for representing knowledge, 1974.
nostalgebraist. interpreting gpt: the logit lens, 2020. URL AckRBssDDddaNGvGru/ini wttp://rrwreieeeeeennnneee teapppreiinnggttttteee pott-the-oggitt11 -een..
Alberto Paccanaro and Geoffrey E. Hinton. Learning distributed representations of concepts using
linear relational embedding. IEEE Transactions on Knowledge and Data Engineering, 13(2): 232-244, 2001.
11

---

Published as a conference paper at ICLR 2024
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNNIP-ICCNLP), pp. 2463-2473, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Richard H Richens. Preprogramming for mechanical translation. Mechanical Translation, 3(1): 20-25, 1956.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5418-5426, 2020.
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back- pook-propagating errors. nature, 323(6889 1986.
Xing Shi, Inkit Padhi, and Kevin Knight. Does string-based neural mt learn source syntax? In Proceedings of the 2016 conference empirical methods in natural language processing, pp. 1534, 2016.
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4222 4235, 2020.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Elena Voita and Ivan Titov. Information-theoretie probing with minimum description length. arXiv preprint arXiv:2003.12295 2020.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. formerrree--ja May 2021.
Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering, 29(12):
2017.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-thh-art natural language processing. arXiv preprint 1rX1v:1910.03771, 2019.
Jinfa Yang, Yongjie Shi, Xin Tong, Robin Wang, Taiyan Chen, and Xianghua Ying. Improving knowledge graph embedding using affine transformations of entities corresponding to each relation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 508-517, 2021.
12

---

Published as a conference paper at ICLR 2024
A RELATIONS DATASET
The dataset consists 47 relations stratified across 4 groups, factual, linguistic, bias, and common- sense. Six of the factual relations were scraped from Wikidata while the rest were drawn from the COUNTERFACT dataset by Meng et al. (2022). linguistic, bias, and commonsense relations were newly curated by the authors. See Table for details.
Table 4: Number of examples per relation and the count of accurate predictions by different LMs. Each of tested using & ICL examples (n 5 for LLaMA-13B) Results presented ICT examples. For cases where the count of examples LRE estimation was not calculated for such cases. ** We also do not calculate LRE for LLaMA-13B where o is year (president birth year and president election year) as LLaMA tokenizer splits years by digits (see Table 9).
the examples std) of the counts across 24 trials with different accurately predicted by LM less then n, it was replaced with
(#:
Correct GPT22xl
Category
Relation
#
GPT-J
LLaMA-13B
person mother
994
182.8 + 5.8
83.5 +: 12.3
613.5 1.5
person father
991
206.1 7.6
109.2 6.1
675.5 + 5.5
person sport position
952
243.3
199.9 1
200.0
landmark on continent
947
421.9 85.3 74.7
2.0.0
person native language
919
320
697.3
2.0.0
landmark in country person occupation company hq
836
14.2
274.1 58.2
2.0.0
821
181.5
42.0
404.0
674
10.
148.8
2.0
product by company person plays instrument
522
366.418.1
262.4 23.8
4.2
513
237.61130.77
17.
219.0 = 67.7
star constellation name
362
2.6
210.3
2.0
plays pro sport
318
24..2
19.11
294.8
company CEO
298
90.2
1.14
1.7
factual
superhero person
100
2.0
3..6
2.3
superhero archnemesis
96
21.
12.7
person university pokemon evolution country currency food from country city in country country capital city country language country largest city person lead singer of band
91
41.0
35.4+1.9
44.8
2.9
3.1
35.7
1.9
2.8
22.0
30
16
1..214
18.8
27
18.0
18.0
18.1
24
1..0
1.6
1..3
0.64
1..9
15.8
24
1.5
0.80
1.5
21
1.2
10.1 0.9
13.0 - 0.0
president birth year president election year
19
1..0
0.5
9.9 0.6
object superclass word sentiment
76
55.7
5.2
5..41.7
60
4..2
42.8
5..
task done by tool substance phase of matter
52
3..6
25.8
33.7
50
3.84
3.3
4.95
commonsense
work location
18.2126
2.6
24.7
fruit inside color
36
10.
9.0
1.6
task person type
32
18.0
1.51
19.2
fruit outside color
30
1..7
9.7
14.6
word first letter
241
4.5
9.9
233.0 0.0
word last letter
241
28.2
2..2
188.3
adjective antonym
100
2.3
2..
6
linguistic
adjective superlative
80
7.9
6.34
7.7
verb past tense adjective comparative
6.6
5..0
65.8
59.5
5..6
60.0
occupation age univ degree gender name birthplace
45
25.7
22.9
3288 2.6
38
2.4
24.2
17.1 2.6
1..0
2.1
hias
name religion
17.0
15.1
19.8 1
characteristic gender
30
15.9 + 2.7
15.8 2.2
19.7 + 1.2
name gender occupation gender
19
11.0 0.0
1.67
10.8 0.4
19
0.8
9.8
10.8 0.4
13

---

Published as a conference paper at ICLR 2024
B ASSUMPTIONS UNDERLYING THE LRE APPROXIMATION
Our estimate of the LRE parameters is based on an assumption that the transformer LM implements relation decoding F(s, c) near-linear fashion that deviates from linear model with a non- linear enror term (s) where both and e' are zero in expectation over s, i.c., Eg [s))]] 0 and
E2 [s[= =
F(s, c) = b+ Ws + c(()
(9)
Then passing expectations over the distribution of we can estimate b and W:
= F(s, c) Ws E(s)
(10)
b E [s, (s.c) - Ws] Bale(s)]
(11)
W (s) es)
(12)
W = [_ c)) BL
(13)
Equations 11 and 13 correspond the bias term b and and projection term W of Equation in the main paper.
C IMPROVING THE ESTIMATE OF F'(s, c) AS 3W
Empirically we have found that 3W (with 3 > 1) yields more accurate linear model of F than W. In this section we measure the behavior of F the region between subject representation vectors to provide some evidence this.
Take two subject representation vectors 81 and $2. The projection term of our LRE model W, based on the mean Jacobian of I' calculated at subjects s;, yields this estimate of transformer's behavior when traversing from one to the other
F(s2) - F(s)) = W(s; s1).
(14)
We compare this an exact caleulation: the fundamental theorem of line integrals tells that integrating the actual Jacobian along the path from s1 to $2 yields the actual change:
F(s2) F(s1)
F'(s)ds
(15)
8 1
[F((s)) F(s:)ll
F')ddss
(16)
Here reduce it one-dimensional problem, defining unit vectors x F(sz) F(s,) and v x S2 81 in the row and column space respectively, so that
|sF(()) F(s:))l uw v||s2 s s1ll
(17)
By taking the ratio between the sides (17) we can see how the actual rate of change from s₁ to 82 is underestimated by W. Table reports this value for some selected relations.
Table 5: Ratio between the right hand sides of Equa- tion (16) and (17) for some of the relations.
Relation
Underestimation Ratio
In practice we find that setting as a constant for an LM (instead of setting it per relation) enough to attain good performance across range of relations. Refer Appendix E for further details.
plays pro sport country capital city
2.517 + 1.043
4.198 0.954
object superclass
3.058 = 0.457
name birthplace
4.328 + 0.991
D CAUSALITY
D.1 DERIVATION OF BQN 7
Under the same relation person plays instrument, consider two different (s, o) pairs: (s Miles Davis, o = trumpet) and (s' = Cat Stevens, o' guitar). If an LRE defined by the projection term and translation term b is well-approximation of the model calculation F(s, c). then
14

---

Published as a conference paper at ICLR 2024
o = (s)) + and o' = =BW(s')
Subtracting o' from
o' BWs' 3Ws
3W (s' s) [since W is linear]
As = s' $ W (o' o)
(18)
We observe that the edit direction As needs be magnified to achieve good edit efficacy. In experiments we magnify As by 3 (or set 3 = 1.0 in Equation (18)).
= (o o)
(19)
D.2 WHY LOW-RANK INVERSE Wt INSTEAD OF FULL INVERSE W-1 NECESSARY?
In practice, we need to take a low rank approximation W instead of W-1 for the edit depicted in Figure be successful. With set to 1.0,
W((' s)
Ao WAs
If we take UEVT the SVD of W, then
Ao UEV As
T Ao ΣV As
Considering U Ao as O. and VT As as Sp
O, = Es, or Σ- 0, = Sp
(20)
Here, maps S, to O,, This E is diagonal matrix that contains the non-negative singular values in it's diagonal and zero otherwise. The greater the singular value the more its effect s,, However, if we take the full rank inverse Σ while mapping o, to sy then the inverse becomes dominated by noisy smaller singular values and they wash out the contribution of meaningful singular values. Thus, it is necessary to consider only those singular values greater than a certain threshold 7 or take a low rank inverse Wt instead of full inverse W-1
The significance of different ranks on causality depicted on Figure 9. We see that the causality increases with rank up to point and starts decreasing afterwards. This suggests that, ablating important singular values from before an optimal rank pr is reached, and start introducing noisy singular values afterwards.
E SELECTING HYPERPARAMETERS (B, ly. AND Pr))
We need to select a scalar value 3 per LM since the slope W of the first order approximation underes- timates the slope of F(s, c) (Appendix C). Additionally, need to specify two hyperparameters per relation r; f,, the layer after which s is to be extracted and pr. the rank of the low-rank inverse W
We perform grid search to select these hyperparameters. For specific 3, hyperparameters E, and Py are selected to achieve the best causal influence as there is strong agreement between faithfulness and causaliry when the hparams achieve best faithfulness there is a weaker agreement between our evaluation metrics (Figure 10). Appendix E.2 provides insight this.
selected this way. However, when f, and p, selected to
15

---

Published as a conference paper at ICLR 2024
Faithfulness
Causality
1.0
1.0
0.8
0.8
Causality
0.6
0.6
Score
0.4
0.4
0.2
0.2
0.0
2 Rank
0.0 0.0
0.2
0.4
0.6
0.8
1.0
Faithfulness
Figure 9: Initially, faithfulness and causality of LRE improve higher rank. However, after rank 2 causality starts declining whereas faithfulness remains stable.
Figure 10: When hparams are selected to achieve best faithfulness there is weaker correlation of 0.74 between our evaluation metrics unlike Figure where hparams were selected to achieve best causality. For both this figure and Figure 6 the LM is GPT-J and 8 == 2.25.
In sweep over layers notice that LRE perfor- mance per relation increases up to a certain layer
and drops afterwards, suggesting mode-switch in later layers (Figure 11). And, in the sweep over ranks we notice that both edit efficacy and faithfulness increases to a certain rank. After that edit efficacy starts dropping while faithfulness remains stable. The reasoning behind how causality is affected by higher rank is discussed in Appendix D.2.
E. SELECTING 3
Table 6: Scores achieved by LRE on different values of / on GPT-J. 3 2.25 shows the best correlation between our evaluation metrics. Faithfulness, means the average faithfulness across all the relations (same for Causality,).
Table 6 represents how performance scores of LRE change with respect to different values of for GPT-J. In our experiments, causality is always calculated with / set to 1.0. So, the average causality
remain constant. /8 is selected per LM to achieve the best agreement between our performance metries faithfulness and causality. For GPT-J optimal value of | is 2.25.
8
Faithfulness,
Causality, Corr
0.00
0.17 = 0.20
0.30
0.25
0.21 0.22
0.33
0.50
0.28 = 0.24
0.38
0.75
0.36 0.25
0.48
E.2 LAYER-WISE LRE PERFORMANCE SELECTED RELATIONS (GPT-J)
1.00
0.43 = 0.26
0.58
1.25
0.50 = 0.26
0.67
1.50
0.54 0.25
0.76
If LRE remains faithful to model up to layer
1.75
0.57 = 0.25
0.80
l fasth it is reasonable to expect that LRE will re- tain high causality until faith well. However, an examination of the faithfulness and causal- ity performances across layers reveals that the causality scores drop before footh (Fig. 11), In fact Fig. 14 from Appendix G.2 indicates that all the intervention baselines exhibit decrease performance deeper layers, particularly our
2.00
0.59 0.25
0.83
2.25
0.59 15990.2. 0.25
0.84
0.81 + 0.22
2.50
0.59 = 0.25
0.84
2.75
0.59 0.25
0.84
3.00
0.59 = 0.25
0.83
3.25
0.58 0.25
0.81
3.50
0.57 = 0.25
0,80
3.75
0.56 + 0.25
0.78
method and the oracle method at similar layers. This might a phenomenon associated with this type of intervention in general, rather than a fault with our approximation of the target s'.
4.00
+ 0.25 0.25
0.76
4.25
0.54 0.25
0.75
4.50
0.53 0.25
0.74
4.75
0.53 # 0.25
0.72
Notice that, in all of our activation patching ex- periments we only patch a single state (at the position of the last subject token after layer E).
5.00
0.52 + 0.25
0.71
the activation after layer is patched, layers till 1 retains information about the original subject
and they leak information about later layers because of attention mechanism. The deeper
16

---

Published as a conference paper at ICLR 2024
intervene the more is this leakage from previous states and it might reduce the efficacy of these single state activation patching approaches.
It is not reasonable expect high causality after foath and causality drop well before fatt because this leakage. This also partly explains the disagreement between the two metrics when the hyperparameters chosen to achieve the best faithfulness (Figure 10).
capital city
food from country
word sentiment
adjective comparative
1.0
1.0
0.8-
0.5
0.5
0.8
0.6-
0.5-
0.5-
0.5
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
Fi
birthplace
religion
work location
task done tool
1.0
1.0
0.5-
0.5
0.5-
0.8
0.6
0.6
0.5
0.6
0.4
0.4-
0.4
0.4
0.2
0.2
0.2-
0.2
0.0
0.0
0.0
company CEO
pokemon evolution
constellation
person mother
1.0
1.0
1.0
0.8
0.5
0.8
0.8
Sone
0.6
0.5
0.5
0.5
04
0.4
0.-
0.4
02-
0.2-
0.2
0.2
0.0
0.0
500
200
Faithfulness
Causality
Figure 11: LRE performance for selected relations different layers of GPT-J. The last row features some of the relations where LRE could not achieve satisfactory performance indicating a non non-lincar decoding process for them.
F VARYING n. AND PROMPT TEMPLATE
Figure 12 shows how lre performance changes based on number of examples n used for approxima- tion. For most of the relations both faithfulness and efficacy scores start plateauing after 5. In our experiment setup we use = as that is the largest number we could fit for a GPTTJ model on single A6000. However, Figure 12 suggests that good LRE estimation may be obtained with less number of examples.
country capital city
plays pro sport
person occupation
object superclass
1.0
1.0
1.0
0.8
0.8
0.8
5core
0.4
0.2
0.2
0.2
0.2
0.0
10
10
10
0.0
10
Faithfulness
Causality
Figure 12: LRE performance across different
We also test how LRE performance relation changes when the same relation is contextualized with different prompt templates. Table shows minimal change in faithuuness and causality scores when LRE is calculated with different prompt templates.
17

---

Published as a conference paper at ICLR 2024
Table 7: LRE performance on different prompt templates. subject s inserted in place O. Performance scores presented as mean and standard deviation across 24 trials with different sets of training examples.
Relation
Prompt Template
Faithfulness
Causality
The capital of O
0.84 0.09
0.94 0.04
The capital of O is the city of
0.87 0.08
0.94 0.04
country capital city
The capital city of Q is
0.84 + 0.08
0.94 1 0.04
what is the capital of ()? It is the city of
0.87 0.07
0.92 - 0.05
O plays the sport of
0.78 0.07
0.90 0.03
plays pro sport
O plays professionally in the sport of
0.78 0.09
0.90 1 0.03
what sport does () play? They play
0.81 + 0.06
0.90 0.03
works professionally
0.41 + 0.08
0.55 0.09
person occupation
O works as
0.44 0.11
0.58 0.07
By profession, Q is
0.46 0.14
0.58 0.08
The superlative form of () is
0.93 0.02
0.97 = 0.02
adjective superlative
what is the superlative form of ()? is
0.92 0.02
0.96 0.03
G BASELINES
G.1 FAITHFULNESS
In Figure 13 we examine whether our method can be applied to extracted from zero-shot prompts that contain only the subject s and no further context. It appears that even when LRE is trained with few-shot examples, it can achieve similar results when applied to that is free of any context specifying the relation.
Factual
Linguistic
Bias
Commonsense
Our method, (sE(s)
LRE (e.) (...................
LineurRegression(s)
TRANSLATION(S) ....
Identity (s)
0.00 0.25 0.50 0.75 1.00 0,00 0.25 0.50 0.75 1.00 0.25 0.50 0.75 1.00 0.25 0.50 0.75 1.00
With fow-shot examples
Without fow-shot examples
Figure 13: LRE faithfulness GPT-J compared with different linear functions baselines(same Figure 4). Each of the functions (Table 2). Dark blue bars indicate faithfulness when evaluated on extracted similar setup. Light blue bars represent how LRE (trained on few-shot examples) generalize when applied on s extracted from zero-shot prompts that contain only the subject and no further context.
approximated with n samples, each prepended with 1 few-shot examples
G.2 CAUSALITY
Figure 14 shows the LRE causality performance in comparison to other baselines for selected relations and across different layers. If LRE a good approximation of the model computation, then causality intervention should be equivalent the oracle baseline, which replaces with s'. The graphs demonstrate the similarity between our method performance and oracle performance across all layers. This provides causal evidence that LRE reliably recover s' for these relations.
While model editing is not the primary focus of this work, it is still worth examining how our intervention may affect multiple token generation, since it may reveal unexpected side effects of the method. Based on a qualitative analysis of the post-edit generations, appears that the edits preserve the fluency of the model. Table 8 presents examples of generated texts after intervention.
18

---

Published as a conference paper at ICLR 2024
work location
lask person type
city in country
person lead singer of band
0.8
Success
0.6
06
0.6
06
04-
0.
0.4
0.4
0.2-
0.2
0.2-
0.2
0.0
0.0
0.0
0.0
Adding W (o'
Replacing with s
Replacing with e,
Replacing with
Figure 14: LRE causality across different layers of GPT-J. The causality curve closely matches the peaks and valleys of the oracle baseline, replacing with s', suggesting that LRE is good approximation of the model computation F.
Table 8: to o'.
Generated texts, before and after our causal intervention on GPT-J to change its prediction
Prompt
-
Before
After
Miles Davis plays the Siri was created by
trumpet + guitar
trumpet in his band. Apple as personal as- sistant.
guitar live with his band. Google and it has be- come a huge success within Google Maps.
Apple - Google
Chris Martin is the lead singer of
Coldplay - Foo Fighters
Coldplay and a man many talents.
Foo Fighters, one of the most successful and pop- ular rock bands in the world. walked. What is the past tense of read?
What is the past tense closed + walked of close? is
closed. It means it gets closed or is closed.
H LRE ON GPT2-XL AND LLAMA-13B
We provide further results for GPT2-xl and LLaMA- 13b to show that autoregressive LMs different sizes employ this linear encoding scheme for range of different relations. Figure 15 present LRE performances for each of the three models grouped by relation category.
1.0
0.8
0.8
Faithfulness
Causality
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
Factual
Linguistie
Commonsense
Blas
Factual
Linguistic
Commonsense
Bias
GPT2-xl
GPT-J
LLaMA-13b
Figure 15: LRE performance in different relation categories on different LMs.
Figures 16a and 16a illustrate the high correlation between our two metrics on GPT2-xl and LLaMA- 13b respectively. These findings consistent with the results reported for GPT-J (Figure 6).
Similarly, Figures 17a and 17b, compare the faithfulness our method with other approaches of achieving linear decoding scheme for GPT2-xl and IlaMA.
19

---

Published as a conference paper at ICLR 2024
1.0
1.0
namo gender
0.8
word first letter
0.8
Causality
0.6
Causality
0.6
word first letter
0,4
0,4
person occupation
task type
0.2
0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Faithfulness
Faithfulncss
(a) GPT2-x1 (8 = 2.25, corr 0.85)
(b) LLaMA- 13B, (3 8.0, corr 0.83)
Figure 16: High correlation between faithfulness and causality in both GPT2-xl (R=0.85) and LLaMa- 13B (R=0.83). Each of the dots represent LRE performance for relation. Bold dots indicate relations for which LRE is evaluated 30 test examples.
With fow-shot examples
Without few-shot examples
Factual
Linguistic
Bias
Commonsense
method, (RE(s)
LRE (e.)
LincurRegression(s)
TRANSLATION(S).
Identity (s)
0.00 0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
(a) GPT2-xl
Factual
Linguistic
Bias
Commonsense
Our method, LRE
LRE (e.) ................................
LineurRegression(s)
TRANSLATION(S) ....
Identity(s)
0.00 0.25 0.50 0.75 0.00 0.25 0.50 0.75 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00
(b) LLaMA-13B
Figure 17: LRE faithfulness GPT2-xl and LLaMA-133B compared with different baselines. Refer to Figure Figure for details these baseline approaches of achieving linear decoding function.
Lastly, Figure 18 depicts LRE faithfulness in GPT2-xl and LLaMA-13b for each of relations in dataset. According to Spearman's rank-order correlation, GPT-J's relation-wise performance is strongly correlated with both GPT2-xl (R = 0.85) and LLaMa- 13B (R = 0.71), whereas GPT2-xl and LLaMa- 13B are moderately corelated (R 0.58).
20

---

Published as a conference paper at ICLR 2024
GPT-J
1.0
Faithfulness
0,7
0.5
0.4
0.1
0.0
GPT2-xl
0.
Farthfulness
0.6
0.5
0.2
LLaMA-13B
0.8
Faithfulness
0.7
0.5
0.4
occupation gender
adjecttvve comparative
adjective past tense
country superlative
largest city name birthplace ace country language country capital city
substance pháse of matter
object superclass lass name religion name gender fruit outside color
characteristic gender plays pro sport adjectivv antonym rocupation age
person native language fruit inside color person university person lead singer of band word sentiment
person plays instrument word first letter country aurrency word last letter landmark on continent
work location president birth year president election year food from country person occupation task person type city in country superhero person position star conste lation name landmark product by company superhero rrchnemesis task done by tool company hg
country
pokemon evolution
person mother!
person father] company CEO
person sport
Figure 18: Relation wise LRE faithfulness to the LM relation decoding function F. Horizontal red lines per relation indicate accuracy of a random-guee baseline. Relations are ordered according to their LRE faithfulness in GPT-J. We do not calculate IRE estimation for GPT2-xl on the relation president birth year GPT2-xl can accurately
for that relation (Table 4). We also do not calculate LRE of I.LMMa11BB for the relations where the o a year (i.e, president birth year and president election year) as tokenizer LLaMA models splits the digits of year. Such behavior of LLaMA tokenizer makes the relation decoding function trivial, since most of the answers start with "I" for these relations. Those cases were included in this plot (grayed out) to align LRE faithfulness of different LMs by relations.
21

---

Published as a conference paper at ICLR 2024
1 LIMITATIONS
Our analysis of linear relational embeddings has several core limitations.
Dataset size We have only tested small set of 47 relations; although we have covered several categories of relations, there are many types of relations we have not explored, such as numerical or physical relations, or logical inferences or multi-hop reasoning.
First-token correctness criterion During all of our experiments, we consider a predicted object correct the first predicted token matches the first token of the true object. This introduces a risk for false positives: if multiple different objects share first token, we might erroneously label prediction as correct. For example, in the relation person university, many of the university names start with "University of" and such cases would inflate our measurements. We quantify the risk for such false positives Table and find that many relations have few to no collisions. Nevertheless, the first-token evaluation scheme limits the relations that can be reliably evaluated for linear decoding using this approach.
Table 9: Relation wise range (count of unique o) in our dataset along with the percentage of os uniquely identified by their first token when tokenized with different LM tokenizers. The number inside parenthesis is the actual count of unique first tokens.
Tokenizer for LLaMA models splits numeric words by digits. For the relation president birth year it only finds unique first token {1), since all the years in this relation start with 1. For president election year LLaMA tokenizer finds only two unique first tokens {1, 2}. Counts for these relations were ignored while calculating average first token coverage for LLaMA-13B.
Relation
rangee
GPT-J
GPT2-xl
LLaMA-13B
adjective antonym
100.0% (95)
100.0% (95)
98.9% (94)
adjective comparative
57
100.0% (57)
100.0% (57)
93.0% (53)
adjective superlative city in country company CEO company hq country currency landmark in country person father person lead singer of band
97.5% (77)
97.5% (77)
98.7% (78)
21
95.2% (20)
95.2% (20)
95.2% (20)
287
72.5% (208)
72.5% (208)
67.6% (194)
163
100.0% (163)
100.0% (163)
93.3% (152)
23
100.0% (23)
100.0% (23)
91.3% (21)
91
100.0% (91)
100.0% (91)
97.8% (89)
968
41.3% (400)
41.3% (400)
38.9% (377)
85.7% (18)
85.7% (18)
85.7% (18)
person mother person occupation
962
39.5% (380)
39.5% (380)
31.9% (307)
31
100.0% (31)
100.0% (31)
93.5% (29)
person university
69
53.6% (37)
53.6% (37)
50.7% (35)
pokemon evolution
90.9% (40)
90.9% (40)
$1.8%
president birth year president election year product by company star constellation name superhero archnemesis superhero person task done by tool
15
60.0% (9)
60.0% (9)
6.7% (1)*
18
77.8% (14)
77.8% (14)
11.1% (2)*
30
100.0% (30)
100.0% (30)
86.7%
31
93.5% (29)
93.5% (29)
87.1% (27)
90
84.4% (76)
84.4% (76)
81.1%
100
89.0% (89)
89.0% (89)
84.0% (84)
51
98.0% (50)
98.0% (50)
90.2% (46)
Relation where all o uniquely identified by the first token (x26)
100%
100%
100%
Average
93.17%
93.17%
92.17%
Single object assumption In some relations, there may be more than one correct answer (e.g.. fruits often take many different outside colors). Our dataset only catalogs one canonical related object in each case. This does not impact results because for each subject we focus on the single object that the LM predicts, but future work could extend our evaluation scheme to measure how well LREs estimate the LM's distribution of candidate objects.
22

---

Published as a conference paper at ICLR 2024
Table 10: Relation-wise LRE performance on GPT-J, and respective hyperparameters.
relation, r
Irange,]
l
B
Pr
Faithfulness
Causality
adjective antonym
95
8
243
0.69 = 0.07
0.86 = 0.04
adjective comparative
57
10
121
0.98 = 0.01
0.94 0.04
adjective superlative
79
10
143
0.93 = 0.02
0.99 + 0.01
characteristic gender city in country company CEO company hq country capital city country currency country language country largest city food from country fruit inside color fruit outside color landmark in country landmark on continent name birthplace name gender name religion
1
74
0.97 0
21
2
115
0.44
0.09
287
6
173
0.05
163
6
126
0.21 0.06
0.49
24
3
0.99 0.07
23
3
88
0.08
0.03
14
63
0.88 0.09
0.99
24
10
74
0.92
0.99
26
3
113
0.51 0.12
0.97
6
7
107
0.65 0.15
0.07
5
160
0.78 0.15
0.12
91
6
97
0.06
0.08
5
4
158
0.16
0.91
7
91
0.05
0.96
2
emb
17
0.16
0.04
4
0.10
0.99
object superclass
10
7
91
0.85 0.05
0.93
occupation age occupation gender person father
5
34
0.68 0.03
1.00
2
4
34
0.98 0.04
1.00
968
8
2.25
217
0.07 0.03
0.28
person lead singer of band
21
8
163
0.64 0.09
0.89
person mother person native language person occupation
962
6
170
0.14
0.09
30
6
92
0.16
0.07
31
8
131
0.00
0.66
person plays instrument
9
198
0.19
0.76
person sport position person university plays pro sport pokemon evolution president birth year president election year product by company star constellation name substance phase of matter superhero archnemesis superhero person task done by tool task person type verb past tense
14
5
97
0.11
0.74
69
4
153
0.14
0.01
6
117
0.06
0.94
44
7
206
0.05
0.25
15
6
106
0.14
0.84
18
emb
84
0.22
0.91
30
4
158
0.11
0.55
31
8
152
0.41
0.27
7
60
0.87 0.09
0.03
90
11
192
0.30
0.10
100
8
228
0.06
0.71
51
5
145
0.10
0.76
32
8
109
0.10
0.17
76
11
182
0.00
0.97
word first letter word last letter word sentiment
25
7
0.09
0.92
18
6
61
0.57
0.83
4
94
0.16
0.93
work location
24
5
112
0.55 0.09
0.94
23


Reviews:
[{"review_id": "8fi4H1byOC", "reviewer": "Reviewer_wBgc", "summary": "This work focuses on analyzing the computation of LLM in the tasks of knowledge decoding. The authors find that a certain kind of computation in relation decoding can be approximated by linear relational embeddings such that R(s) = \\beta Ws + b. Specifically, the intermediate hidden representation of subject is used in this linear transformation. With experiments on 47 different relations, the authors find that for some relations the linear approximation hold. However, this linear attribute is not universal. Furthermore, the authors conducted experiments to show the causality of the linearity of relation decoding. Finally, a visualization tool called attribute lens is proposed to show where and when the LM finishes retrieving knowledge about a specific relation.", "questions": "In term of handling objects that have multiple tokens, I saw in Table 4: only the first token was used to determine correctness. Do we apply this strategy to all experiments? Will this lead to false positive? For causality experiment, do we also use the first token of o’ for experiment?\nDo we have some examples showing that `when LRE can not fully capture the LM's computation of the relation, the linear approximation can still perform a successful edit`?\nIn which level of faithfulness, we can say that the linearity holds for the relation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This work focused on an interesting question about what computations the LMs perfrom while resolving relations. The authors smartly use the local derivative to obtain the affine transformation approximation. This is aligned with the traditional design in training knowledge graph embedding. Also, it is very useful to show that causality of the linearly decoding behavior, by using a low-rank pseudoinverse to obtain the perturbation of subject.", "weaknesses": "It is unknown that how the context prompt (e.g., [s] plays the) affects the conclusion. For example, will the conclusion hold if we change some other contexts that express same meaning of relation? Though it is interesting to show that some relations matches the linearity hypothesis (e.g., occupation gender, adjective comparative) which aligning traditional methods of training knowledge graph embedding, my concern is that the faithfulness is not very high for most of relations. This paper is more like a case study instead of a systematic measurement which covers a broader range of relations. It is hard to conclude how much percent of relations match the hypothesis. \nFurthermore, it is unclear, if I did not miss some texts, why the causality is usually higher than faithfulness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TZxVxRXMVI", "reviewer": "Reviewer_y12e", "summary": "This paper analyzes how the knowledge of relational triples (i.e., subject-relation-object) can be decoded in large language models (LLMs). Specifically, the authors hypothesize that the object embedding can be computed using an affine function (i.e. LRE) from the subject embedding and show that this holds for the majority of relations tested in this work. They also show the causality of the relationship through intervention experiments where a different object can be obtained by changing the subject embedding according to the LRE.  The authors also present the Attribute Lengs, an interesting application of their observation similar to the Logit Lens, which allows one to examine what entities are predicted to be the object in intermediate layers of a transformer model.", "questions": "- In experiments, n = 8 examples are used to estimate W and b. Would it be difficult to use more examples?  They seem to be too few to reliably estimate W and b.  I would also be interested in the variance as well as the mean.  \n- Does a “sample” mean a single example?  In statistics, a sample usually means a collection of examples (data points).\n- p. 4: of LM’s decoding -> of the LM’s decoding?\n- p. 5: by (Merullo et al., 2023) -> by Merullo et al. (2023)?\n- p. 7: is a higher -> is higher?\n- p. 8: visualizes a next -> visualizes next?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper presents a novel insight into how relational knowledge is represented in an LLM, which should contribute to a deeper understanding of LLMs in the field.\n- Extensive experiments are carried out to confirm the findings.\n- The paper is well written and easy to follow", "weaknesses": "- The configuration for estimating the parameters of the affine functions could be further explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GdRn1xbN8d", "reviewer": "Reviewer_zSUm", "summary": "*update* The authors have addressed most weaknesses/errors that I have raised. I updated my score upwards as a result. */update*\n\nIt is well known that language models acquire knowledge of subjects, objects and their relations. However, to date, it is not well understood how these relations are represented in the model. The paper posits that relations are implemented as Linear Relational Embeddings, that is, affine transformations mapping from the embedding of the subject to the embedding of the object. The paper proposes to compute these mappings from the Jacobian $\\delta o / \\delta s$, where o and s are object and subject embeddings at certain layers, respectively.\nThe paper validates this approach by measuring a) faithfulness, i.e., whether decoding from the LRE mapping matches decoding from the true model when given a prompt expressing the relation, and b) to what extend this mapping is a causal explanation by providing an inverse of the LRE capable to change s such that a different, desired o results.\nThe experiments show that the method results in relatively high, but far from perfect, faithfulness and causality scores. The method is compared to number of reasonable baselines/ablations. Finally, the method is applied to the use case of detecting when the language model outputs false relationships despite knowing the true relationship.", "questions": "Please respond to the weaknesses raised. I am happy to raise my score if the explanations are satisfactory.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "* the paper covers an important topic that will certainly raise interest among the ICLR attendees\n* the methods and results are interesting - I learned something.\n* the presentation is excellent. The paper is easy to follow and results are presented in a comprehensive fashion.", "weaknesses": "* Limitations are not discussed in enough detail. For example, given a fixed subject and relation, it is possible that there are multiple true objects. This has consequences both for the generation of the datasets (where examples are filtered out if they're not generated by the model) and questions to what extend an invertible function can be a reasonable approximation of the true function, since invertibility implies that the function/relation is injective. Discussing these aspects would improve the paper.\n\nI am a bit skeptical with regards to some results:\n* Table 4 (Appendix A) shows the number of correctly predicted relations per language model. It is quite remarkable that some models in some categories get zero percent of the facts right, whereas other models get a much larger percentage right. Zero percent hints at systematic errors that remain undiscussed in the paper. For example, the LLaMA-13B achieves zero percent on the task of predicting president's birth year and election year. This is quite curious, given that these are quite well-known facts that appear a lot in the pretraining data, and the other models get them right. There is a reason for the poor performance of LLaMA on this particular task: Their tokenizer represents every digit as its own token. However, the caption of Table 4 states that only the first token was used to determine correctness. Since the LLaMa model would require 4 tokens, it had no chance of predicting the \"correct\" token. IMO, this is shortcoming of the proposed measurement, and accepting this bias should at least be discussed. I suspect that similar reasons might cause zero percent accuracy on some of the other tasks.\n* Table 4 (Appendix A) shows that GPT-J gets zero examples from the \"occupation gender\" category right. Hence, none of these examples from this category should remain in the dataset for GPT-J according to the description in Section 4 (Dataset). Curiously, Figure 3 shows that \"occupation gender\" has a near 100% faithfulness success rate. How can that be if there are no such examples?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)