Unveiling LLMs: The Evolution of Latent Representations in a Dynamic Knowledge Graph
Authors: Marco Bronzini
Carlo Nicolini
Bruno Lepri
Jacopo Staiano
Andrea Passerini
Venue: COLM (2024)
Keywords: Large Language Models, Latent Representations, Factual Knowledge, Activation Patching, Graphs, Knowledge Graphs
TL;DR: A framework based on the technique of activation patching to represent the factual knowledge embedded in the vector space of LLMs as dynamic knowledge graphs.

Abstract: Large Language Models (LLMs) demonstrate an impressive capacity to recall a vast range of factual knowledge. 
However, understanding their underlying reasoning and internal mechanisms in exploiting this knowledge remains a key research area.
This work unveils the factual information an LLM represents internally for sentence-level claim verification.
We propose an end-to-end framework to decode factual knowledge embedded in token representations from a vector space to a set of ground predicates, showing its layer-wise evolution using a dynamic knowledge graph.
Our framework employs activation patching, a vector-level technique that alters a token representation during inference, to extract encoded knowledge.
Accordingly, we neither rely on training nor external models.
Using factual and common-sense claims from two claim verification datasets, we showcase interpretability analyses at local and global levels.
The local analysis highlights entity centrality in LLM reasoning, from claim-related information and multi-hop reasoning to representation errors causing erroneous evaluation.
On the other hand, the global reveals trends in the underlying evolution, such as word-based knowledge evolving into claim-related facts.
By interpreting semantics from LLM latent representations and enabling graph-related analyses, this work enhances the understanding of the factual knowledge resolution process.

Published as a conference paper at COLM 2024
Unveiling LLMs: The Evolution of Latent Representations in Dynamic Knowledge Graph
Marco Bronzini*), Carlo Nicolini Bruno Lepri?.3, Jacopo Staiano", and Andrea Passerini
Uniiersitty of Trento, Trento, Italy 2Ipazia S.p.A., Milan, Italy
Fondazione Bruno Kessler (FBK), Trento, Italy
Abstract
Large Language Models (LLMs) demonstrate an impressive capacity to recall a vast range factual knowledge. However, understanding their un- derlying reasoning and internal mechanisms in exploiting this knowledge remains a key research area. This work unveils the factual information an LLM represents internally for sentence-level claim verification. We pro- pose an end-to-end framework to decode factual knowledge embedded in token representations from vector space to set of ground predicates, showing its layer-wise evolution using a dynamic knowledge graph. Our framework employs activation patching, vector-level technique that alters a token representation during inference, to extract encoded knowledge. Accordingly, neither rely on training nor external models. Using factual and common-sense claims from two claim verification datasets, we show- case interpretability analyses local and global levels. The local analysis highlights entity centrality in LLM reasoning, from claim-related informa- tion and multi-hop reasoning to representation errors causing erroneous evaluation. On the other hand, the global reveals trends in the underlying evolution, such as word-bas -bsed knowledge evolving into claim-related facts. By interpreting semantics from LLM latent representations and enabling graph-related analyses, this work enhances the understanding of the factual knowledge resolution process.
Introduction
Several studies have demonstrated the ability of Large Language Models (LLMs) store and recall an impressive variety of common-sense and factual knowledge (Meng et al., 2022; Jiang et al., 2020; Shin et al., 2020; Brown et al., 2020; Petroni et al., 2019). However, investigating how LLMs leverage this knowledge and their reasoning remains an ongoing research challenge. This work studies LLMs' knowledge resolution mechanism and represents its underlying evolution dynamic knowledge graph. It addresses three research questions: (i) Which factual knowledge do LLMs use to assess input truthfulness? (ii) How does this knowledge evolve across layers? (iii) Are there any distinctive patterns in its evolution?
We investigate how factual knowledge, encoded in the latent spaces of LLMs, changes during inference when tasked with claim verification. Specifically, we propose a framework to reveal the factual information LLM holds internally when evaluating the truthfulness of short claims such as "Charlemagne was crowned emperor on Christmas Day". " It unveils non- trivial insights into the internal workings of LLMs exhibited Figure 1. Analysing the vector space of LLMs, also known as latent representations, implies tracking the evolution of token representations across the model's hidden layers (Vaswani al., 2017) and segmenting inference into discrete time steps: layer at time f, layer + 1 at time f + 1, and so forth.
*Corresponding author. marco.bronziin 10unitt.it
'The framework and its code available Python package named Latent-Explorer.

---

Published as a conference paper at COLM 2024
Falise
Input sentence
LLM (Claim verification)
CHARLEMAGNE
Charlemagne was crowned emperor on Christmas Day True
12/25
DIRSTIAS DUY
Subject's factual information
Representation errors
Multi-hop reasoning
MODEL OUTPUT
800
814
42
From HIDDEN LAYER
From HIDDEN LAYERTE! From HIDDEN LAYER
Figure 1: Illustrative insights from unveiling the process of factual knowledge resolution within an LLM using the proposed patching-based framework.
Here, we develop framework (Figure 2) to jointly decode factual knowledge embedded in LLMs' latent representations and represent its dynamics using a graph representation. Initially, we elicit model's s behaviour by assigning a language model (LLaMa2; Touvron et al., 2023) the task of verifying entire claims, while storing token representations during inference. Next, we prompt a separate model inference (Figure 2) to decode the semantics embedded in these representations using activation patching (Zhang & Nanda, 2023). Since patching is a token-level technique, we beforehand define function that maps the input claim's token representations to vector representation. This matrix-to-vector mapping conderepresenn function creates a summarised and condensed representation of the input using a weighted sum, exploiting the additive property of token representations to combine their embedded semantics. It offers an alternative to multi-token patching, which treats each token inde- pendently and extends single-token patching (Ghandeharioun et al., 2024). Afterwards, the inference patched with the input's summary interprets its encoded semantics as ground predicates such as IsDate (Christmas Day, December 25). This formalisation has two ad- vantages: (i) formatting factual knowledge consistently for subsequent knowledge graph generation and (ii) connecting information using logical symbols (A and -). Lastly, we represent the extracted knowledge, and its evolution, as dynamic knowledge graph (Fig- ure 2), combining multi-relational graph of entities and relations (Wang et al., 2017) with dynamic graph² (Harary & Gupta, 1997). This allows us to visually represent the process of factual knowledge resolution using the model's layers the graph's temporal dimension.
After collecting factual and common-sense claims from two well-known claim verification datasets, FEVER (Thorne et al., 2018) and CLIMATE-FEVER (Diggelmann et al., 2020), we showcase two use cases for the outputs of the proposed framework: local and global interpretability analyses the factual knowledge decoded from latent representations. Local interpretability highlights knowledge centrality in LLM reasoning: from the subject's factual information and multi-hop reasoning to representation errois causing erroneous evaluations (see Figure 1). On the other hand, the graph representation helps reveal global trends in LLMs' factual knowledge resolution process, from middle-layer importance to word-based information evolving into claim-related facts. Our main contribution is an end-to-end framework that jointly accomplishes several tasks:
decoding the semantics embedded in the latent space of LLMs, in the form of ground predicates; without relying on external models or training processes;
extending single-token patching by exploiting the additive property of LLM's token representations to probe the semantics of multiple tokens jointly;
representing the encoded factual knowledge and tracing its underlying evolution using graph representation;
Where nodes (entities) and edges (relations) change over time.
2

---

Published as a conference paper at COLM 2024
enabling interpretability analyses at both global and local levels, revealing, for in- stance, word-based knowledge evolving into claim-related facts and representation errors that cause incorrect evaluations.
By decoding semantics from LLM latent representations and enabling graph-related anal- yses, this framework advances our understanding of the factual knowledge resolution process and the mechanistic interpretability of language models.
The rest of the work is organized as follows. Section reviews related literature, Section details our approach This is followed by Section outlining our experiments, and Section that presents concluding remarks.
Source prompt &
BIA [I....,]]
M
Target prompt T
Subj
Subj
M
Ob
Oej
Ob
Obi
H
Generated text (inference)
Hidden states R]
Subj
Patching
(o))
Ob
Figure 2: The patching-base framework decodes the factual knowledge from LLM latent representations. The outputs are represented in dynamic knowledge graph.
Related Work
Activation patching is traditionally applied in mechanistic interpretability to study compo- nents and internal workings of machine-learre models. Conventional workflows involve eliciting a behaviour to observe, discovering patterns via activation patching and generating functional hypotheses (Conmy et al., 2024).
Specifically, the patching technique studies the model's computation by altering its latent representations, the token embeddings in transformer-basse language models, during the inference process (Ghandeharioun al., 2024). For example, Meng al. (2022) focused on causal tracing by replacing the latent representation of noise-corrupted inference with the correct ones, identifying the crucial activations in rectifying the correction. Ghandeharioun et al. (2024) proposed an LLM-based framework for semantically inspecting the latent repre- sentations of LLMs. The authors studied the entity resolution process across the early layers of LLMs by patching the latent representation of subject entity (e.g., "Alexander the Great") into separate inference tasked to describe it. Alternatively, Yang et al. (2024) studied LLMs' multi-hop reasoning via framework projecting latent representations They examined LLMs' handling of first-hop and second-hop reasoning tasks in completing two-hop factual propositions, assessing latent bridging entity representation for connecting information fragments. Teehan et al. (2024) proposed instead a framework to generate high-quality latent representations for new concepts using small number of example sentences or definitions. These studies, along with others conducted recently (Mallen et al., 2023; Hernandez et al., 2023; Jiang et al., 2020) have investigated the factual understanding of LLMs regarding single entities in scenarios involving knowledge completion, such as incomplete triplets. On the contrary, our research focuses on entire sentences, exploring the extensiveness and evolution of factual knowledge embedded within LLMs when tasked to evaluate the input truthfulness. This represents a step towards understanding the LLMs' factual knowledge resolution process rather than factual knowledge of single entities.
vocabulary space.
3

---

Published as a conference paper at COLM 2024
3 Methodology
Following the framework proposed by Ghandeharioun et al. (2024), we leverage the ac- tivation patching technique to decode the semantics, in the form factual information, embedded within an LLM vector space and represent its evolution across the model's layers using a dynamic knowledge graph. The procedure outlined in Figure 2.
Given language model M with I. hidden layers, and prompt S containing an input sen- tence, we probe the tokens' latent representations obtained at each hidden layer E [1, during the inference of M on S (residual stream). We execute a separate inference of the model M with different prompt T to decode the semantics embedded in these represen- tations. This prompt T contains a special placeholder token "x" for the patching process: substituting its latent representation with summary of the input's latent representations from the original inference. We specifically intervene during the model computation on prompt T by mapping the placeholder token's embedding with a weighted sum of the token embeddings of the input sentence obtained at the 1-th hidden layer of the model inference on S. The execution of this patched inference generates then a structured text o', containing a list of factual information. The procedure is repeated for the different values of I E []] All the generated texts are then represented as dynamic knowledge graph, with being the graph's dynamic and temporal dimension.
Essentially, the proposed framework converts the LLM internal vector space into human- understandable semantic graph by collectively probing the encoded semantics of multiple token representations. The following further details the different procedure steps.
3.1 Prompt Definition
Initially, we define template for the model instruction, the source prompt S, encompassing three different semantic parts: (i) a system instruction describing how to accomplish the claim verification task, (ii) an input-oulput example to help the model generate the desired output, and (iii) the input sentence (Figure 3). The desired output is structured text with two attributes: (i) a binary label indicating the truthfulness of the sentence, and conjunction (ii) a list of facts supporting such evaluation. The facts are represented of ground literals: asserted or negated predicates. For instance, the factual information necessary to evaluate the input claim "Edgar Allan Poe wrote Hamlet" can be represented as: Authorof (Hamlet, William Shakespeare) A -AuthorOf (Hamlet, Edgar Allan Poe), where and - indicate the logical conjunction and negation respectively. Using ground literals has two advantages: (i) formatting factual knowledge consistently for the subsequent knowledge graph generation, and (ii) stimulating the language model associate and contrast factual information using its logic symbolic knowledge (De Smet et al., 2023). Our preliminary experiments indicated that using a simple bubjectpreddccat-objeee representation leads to sub-optimal outcomes, resulting in more isolated and subject-focused
(SPO) triple
The terms latent/oector representaaiions/spac, embeddings, and hidden states are used interchangeably.
SOURCE PROMPT S
<>*[NNS]] <xSYS>> You are a journalis with experfire facl-hhecknng Your role is to evoluate the trutriiness of factual claims. To uphold journaistic integrity. you must produze a report containing <x/SSS>>
Generated true/false "FACTS
CLAIM DINST] (
ReieiionOOijed, Subject), Relatton(Djjeet Subject)
SINPUT
"LABEL": truertalse,
*FACTS"
Factual claim
Oejest Subject)
Oijecd, Subject) Relstion(Objeed Subject),
Reistion Oqqedt Subject)
<ip>
(?) Hidden states
5ONP17 
]
Figure 3: Inference of M on the source prompt S.
4

---

Published as a conference paper at COLM 2024
TARGET PROMPT T
Vie (l....,
<>>
 

You are an assisiant with experiise in fedi-checking, Your role is to assess claims using zeroth-order logic proposilions. 

Input embedding
PATCHING
127
h
CLAIM [JNST] ('label: trus/false, "facts": [...]|+/s>>
<>>
 CLAIM [MNST] ('label": true/false, [facts": [..)<<>>>>
<>>
 CLAIM (INST] ("label": trueffalse, "facts": "aacis:[---]

.101
Logit
||

 [*NSINT] x [
Generated text (o*)
Figure 4: Patching operation during the inference of model M on the target prompt T.
information.
The full prompt is shown in Appendix A. We then apply this template given input sentence and run the language model M on S, producing structured output and storing the hidden states (H'), the token representations across the model's layers.
3.2 Patching
We denote the matrix H' as the latent representations obtained from the /-th hidden layer of the model's inference on the source prompt S. We then consider separate inference of the model M on a different prompt, a target sequence of m tokens T = (t,, fm). This prompt mimics the source prompt to generate similar outputs but serves decodes the semantics within the latent representations via activation patching. To perform the patching operation,
propositional probe. It
include placeholder token, the character "x", within this prompt. We also augment the in-context examples (1 to 3) and reduce the system instruction to boost the model's in-context abilities during inference. The full prompt shown in Appendix B. This patching operation consists of replacing the vector representation of the placeholder token (hit where k is the position of in T) with a summary of the vector representations of the input sentence from original inference (M on S), leaving the other latent representations unchanged, and letting M proceed with the inference (Figure 4). Since activation patching is vector-level technique, we formally define natrix-to-vector mapping function f(H', W) Raxd + Rd parameterized by W, that computes weighted sum of the latent representations of the input part I C S of the source prompt:
f(H W)
Σ w
w =
(1)
where the input part refers to the input sentence, the factual claim, and L = [Z] is its length. We set the weights W of the weighted sum by performing part-of-speech tagging* on the input claim. Nouns and verbs are assigned weight equal zero to all but their end token which receives a weight of one (Figure 5), emphasising sentence's entities and predicates in this summary vector representation. We focus on end tokens based on Meng al.' 's (2022) study, which found that the model forms a subject representation at the final token of an entity name. Appendix G shows that including all tokens, especially stop words, is detrimental. Additionally, empirical experiments showed that using single tokens leads to meaningless texts for punctuation and single-word information for the last input token.
Patching is then applied by replacing lij with in the model's input embedding layer, as visually exhibited in Figure 4. This model inference, patched with the summary input representation from I-th layer, generates a structured text o', structurally equal the one from the original model inference (M on S). By applying this procedure for all values of I EE [1, we produce structured outputs using all latent representations of M.
whtps://sppayio/mmde////

---

Published as a conference paper at COLM 2024
[SNS] Charlemagne was crowned emperor Christmas Day 

Σ
h
0
Figure 5: Example of the binary weights for the input's tokens I C S. These weights are then used to combine the tokens' vector representations weighted sum.
3.3 Knowledge Graph Generation
We use a knowledge graph to represent the list of ground literals, the factual information, included in the structured output (Figure 3) generated by each patched inference (o' E O M on T) and the original inference (M on S). We first turn literals into subject- predicate-object (SPO) triples using simple rewriting rules:
(-) (-JisPrrpertyyobjjet) + (object, in(ot), property)
+ (object1, rebareet for binary predicates for unary predicates
(2)
Afterwards, we represent all the triples, yielded from o', as knowledge graph (Figure 6). We eventually concatenate all the graphs generated for the different values of I, creating dynamic graph that exhibits the factual knowledge evolution across the model's layers.
HOLEWAY
Label: true
Facts:
ONIA
CHRISTIGG
ChhteeCh Day, December 25 IaHoidayyChiistma
JAN JARLARRY
E1G
Cafe Dealh
CHAR EMAGN
ChareeChhreeaagne, Emperor) A DaleOrBirt Apri 742)
A Uinnortsahiihhaeemmm.... 28 January 814)
SUPEROR
25
Figure 6: Graph generation process from the ground literals knowledge graph. This process is performed for each of O and the inference output.
4
Experiments
This section showcases our framework on two experimental use cases: local and global inter- pretability analyses of the extracted factual knowledge. After describing our experimental setup in Section 4.1 and evaluating the effectiveness of the language model in Section 4.2, we present two interpretability analyses, a local analysis of the factual information of three dis- tinct input claims in Section 4.3, and global analysis of the patterns in the factual knowledge resolution process Section 4.4.
4.1 Experimental Setup
To analyse the model behaviour in recalling factual knowledge to support or debunk input sentences, we prompt a language model with collection of factual and common-sense claims sampled from two well-known claim verification datasets: FEVER (Thorne et al., 2018) and CLIMATE-FEVER (Diggelmann et al., 2020). As in conventional mechanistic interpretability workflows (Conmy et al., 2024), these inputs serve to elicit a model behaviour, unfolding the factual information LLM holds internally (Section 1).
Each claim is labelled as Supported (True), Refuted (False) or NotEnoughInfo. For instance, the claim "Charlemagne was crowned emperor Christmas Day" from the FEVER dataset is classified as Supported ted, whereas "Berlin has a population of 4 million people" is classified as Refuted. These are all factual claims and language model should rely specific factual

---

Published as a conference paper at COLM 2024
knowledge to evaluate their truthfulness. On the other hand, CLIMATE-FEVER, dataset mimicking the FEVER methodology, compounds to real-world claims regarding climate change (Diggelmann et al., 2020). Its claims may require more common-sense reasoning, and elicit subjective dichotomies. For example, the claim "Global warming is increasing the magnitude and frequency of droughts and floods" is labelled as Supported, whereas "Renewable energy investment kills jobs" is classified as Refuted.
We initially filter the two datasets by (i) excluding the claims labelled with NotEnoughInfo, avoiding prompting the model with unverifiable sentences, and (ii) considering claims neither too short nor too long (35 < [characters] < 120). Afterwards, we randomly sample 1,000 claims from FEVER (Supported: 726, Refuted: 274) and 400 from CLIMATE-FEVER (Supported: 274, Refuted: 126). We use the LLaMA language model (Touvron al., 2023) in its 7-billion instruction-tuned version to showcase our framework. Appendix F compares the output generated by similar language models: the model's 's 13-billion version, the newest LLaMA 3 (Al@Meta, 2024) and the 7-billion Falcon model (Almazrouei et al., 2023).
4.2 Classification Performance
Method. We here assess the effectiveness of the language model M to classify the input sentences. This helps to better understand whether the desired model behaviour is properly elicited: recalling helpful knowledge to evaluate claims' truthfulness effectively. The classification metrics consider the binary label within the structured text generated during the inference process (Figure 3, M on S). Table displays the metrics grouped for each target binary label: true (supported) and false (refuted). It also exhibits a self-consistency metric that measures the average consistency of the inferences' binary labels with those generated by the patched inferences (o' E EO M on
Findings. The model achieves good performance in both datasets, reaching ROC AUC scores equal 0.68 and 0.74 for FEVER and CLIMATE-FEVER respectively (Table 1). Table reports also the Fl score. However, examining the model's performance reveals significant imbalance, especially on the FEVER dataset. has high precision and low recall for the claims classified as true, and low precision and high recall for those classified as false (Table 1). Intuitively, this may be because falsifying claim often requires less factual knowledge than is needed to prove it true. On the other hand, the CLIMATE-FEVER dataset demonstrates balanced recall performance for both classes, with a precision reduction in the false ones. This suggests the model encounters comparable difficulty in verifying or debunking claims when this mainly depends on common-sense reasoning (Section 4.1) while erroneously classifying more claims false. The confusion matrices are reported in Appendix D. A dichotomy between the two classes is also observed in the self-consistency metric: when the inference generates true label, on average, almost half of its hidden layers generate the same binary label, whereas just about ten percent of the layers' labels coincide with the inference prediction for the false label (Table 1).
DATASET
TRUE
PRECISION FALSE
AVG
TRUE
RECALL FALSE
AVG
TRUE
FALSE
F1
AVG
ROCAUC
ACCURACY
SELF-CONSISTENCY
IRUE
FALSE
PEVER
01932
11388
0.783
0.456
11912
0.981
0612
11.45
0.594
11684 0.739
01.81
0.54 0.2
0.14 + u1 0100 2 U.1
CLIRALE-HEERR
03873
1L382
0.722
0.703
11722
0.71
0.28
11625
0.31
(121
0.89 0.2
Table 1: Classification performance of the language model the input sentences. AVG indicates the weighted average performance.
4.3 Local Interpretability
Method. We investigate the factual information encoded within the model's latent repre- sentations of three distinct input claims. Claim A ("Renewable energy investment kills jobs") ") from the CLIMATE-FEVER dataset, and requires common-sense reasoning. In contrast,
>https:/
http:////hggnngac...oo/ meta-llama/ 1aamm--27b-chat-hh Receiver Operating Characteristic Area Under the Curve
7

---

Published as a conference paper at COLM 2024
claim B ("Charlemagne was crowned emperor on Christmas Day") and claim C ("George Lucas founded a visual effects company") ") are from FEVER and require multi-hop reasoning. Figure exhibits five snapshots of the dynamic knowledge graphs related to these three inputs, while Appendix C shows the comprehensive graph for claim B.
Ranawabble www.gy. wnw.xiimniieimmmnn...........m.i killa jobs.
LOWE TrNe
LAVK
Jarcn firw
LISES Sganan
FALSE
SOOA
TRUE
Charlemaanaa drowned emerrar Ohrittma Day
LIVER Jomes
Goorge Luces founded a visiual wffecte
LAVES 29/N
Figure 7: Evolution of the factual knowledge decoded from the LLM latent representations. exhibits five snapshots from the dynamic knowledge graphs created for three different input sentences. A colour gradient highlights the temporal progression of the model's inference. The white in the rightmost graph indicates the output from the original inference.
Findings. All claims in Figure show word-based factual information in the early layer L3. However, while claims B and C accurately present information about their entities, claim A incorrectly exhibits information about the last in-context example (The Beatles, Appendix B). In Section 4.4, we hypothesise that example leakage may occur when the probed latent representation lacks factual knowledge, shifting the model's focus the provided context. Claim A then unfolds an interesting knowledge evolution: Layer 7 encodes factual knowledge concerning Energy, whereas Layer 17 and Layer create clear dichotomy between entities that are Job Killer (e.g., Coal and Nuclear Energy) and Job Creator (i.e., Energy Efficiency Upgrades). Although this can be common-sense reasoning, it might unveil biases in this energy-related dichotomy. On the other hand, Claim unveils a swap in the entity representation in the middle layers. Layer correctly encodes factual knowledge concerning the subject entity (Charlemagne) Layer 14 then confuses and swaps it with Charles V (another former Holy Roman emperor), yet succeeds in multi-hop reasoning by connecting Christmas Day, its actual date, and the emperor's 's coronation. The third claim exhibits another error related to internal knowledge representation. Layer correctly encodes George Lucas as Founder Of the company Industrial Light Magic, while treating its acronym (ILM) as different entity. Subsequently. layer 19 associates the information of being special effects company only with the acronym, while separately connecting its full name to its founder, George Lucas. This disjoint association and entity duplication lead to a multi-hop reasoning error in the inference (Figure 7). While unifying the company's is representations, the model fails to reconnect George Lucas as its founder. This Lnowledge-recaling issue might stem from attention mechanisms or catastrophic forgetting.
4.4 Global Interpretability
Method.
This analysis reveals patterns in the evolution of factual knowledge. We seek behaviour changes across the hidden layers, thus identifying transaction points in this latent evolution. We examine the knowledge graphs generated by the latent representation
8

---

Published as a conference paper at COLM 2024
from each hidden layer by calculating graph-level similarities and identifying cross-layer similarities. We initially compute node embeddings for each dynamic knowledge graph derived from subset of over five hundred FEVER dataset inferences. We consider input claims with character lengths falling within the first and third quartiles. We use the Multi- Scale Attributed Node Embedding method (MUSAE, Rozemberezki et al., 2020; 2021) to generate node embeddings Z R for each graph, where и represents the number of graph nodes and d is the embedding dimension (d 4096). Next, we assess the semantic correspondence between two graphs (G and G') using custom graph similarity measure on the node embeddings: (i) computing pair-wise node similarity, (ii) identifying the highest similarity matching for each node of G, and (ii) averaging these similarities:
1 NC L NG
sim (G, G')
max simcos (Z((),,,(()') iE[Lae
(3)
where Hg and f'g' are the number of nodes G and respectively, and sitfioos the cosine similarity. To investigate patterns in the evolution of the encoded factual knowledge, we focus on the similarities of the graphs generated using the latent representations two consecutive layers: and of Appendix provides layer-wise cosine similarities. We repeat this process for all the dynamic knowledge graphs and report aggregated results in Figure 8. Afterwards, we employ the MeanShift clustering algorithm' (Cheng, 1995; Pedregosa et al., 2011) on the similarity data. The bandwidth hyper-parameter, which affects the cluster granularity, estimated using the 25th percentile of the sample data. This clustering procedure identifies layers exhibiting similar changes in the evolution of their factual knowledge (colours in Figure 8), indirectly spotting transaction points.
Similarity the previous layer
120%
30%
60%
bn
122 11p
15
Lig
Up
Lgg
Ly
1200
1zz 1gs 1/4
1gr
100
Graphs from the I-th layer
Figure 8: Cosine similarities between the graph representation of the factual knowledge decoded from the latent representation of the 1-th and the 1-1-th hidden layer. The colours map the layer clusters discovered using the MeanShift clustering algorithm.
Findings. We validate the transaction points unfolded in Figure 8 by examining the type of factual information decoded from the cluster items across several individual inferences (Appendix E). We observed that early layers (L₃ La, cluster B) focus on the syntax and entity resolution process, as already demonstrated by Ghandeharioun et al. (2024). We also discovered slightly different evolution pattern when tokens represent partial or complete words. When tokens correspond to complete words, such as "Germany" and "Robin" (Ap- pendix E), the model encodes factual information related to the entity its semantic context, for instance, IsCountry (Mexico) and SuperHeroof (Robin, Batman). However, in the case of incomplete words, the model accomplishes entity resolution using encoded word-based representations. For example, merging the token representations of "jack" and "eman", the model tries to decode the subject entity as "Jackman", yet the original subject was "Bojack Horseman" (Appendix E). Further, these early layers produce valid outputs (structured texts) in only 40% of the considered inferences. In contrast, the latent representations from the 4th hidden layer (cluster C) generates valid factual information regarding the subject's semantic context in 75% of the inferences. For instance, WasQueen (Mary Queen of Scots) for the
https//aaaaeooooee//ooeosa http://ggeereeeeeeeeeee www.eeeam...uste..eeeee.ee MeanShiif

---

Published as a conference paper at COLM 2024
original subject "Empress Matilda" or AististoriclFFiiuu(CChaaar for "Charlemagne" (Appendix E). Thus, we empirically show that the model exhibits minimal to no factual knowledge in these early layers.
Middle layers (Ls L21, cluster D) exhibit comprehensive factual knowledge concerning the subject entity, with evolution towards the requested factual information, for example regarding whether Empress Matilda moved to Germany (claim in Appendix E): L,= (..., BornIn(Empress Matilda, England)), ), L1s = (..., DEmpIInE Matilda, England) and Lis = {..., MovedTo (Empress Matilda, Germany) J. These middle layers also exhibit an interesting pattern: whenever the model achieves comprehensive factual knowledge for given entity, it moves its attention towards another entity in the sentence (or a semantic entity neighbour), and represents other contextualised factual information. For claim 2 in Appendix E, for example, the model represents the factual information concerning "Robin" " in layer 4, then concerning "The Joker" between layers 6 and 17, and eventually further factual information concerning "Robin" between layers 18 and 19.
On the other hand, we noticed decline the factual knowledge decoded from the latent representations of the bottom layers (L27 L32, cluster A) as well as the initial layer. Over 90% of the inferences patched with these layers yield unstructured texts, often containing nonsensical text references to the last in-context example (The Beatles, Appendix B). The latter leads us to speculate that the model's attention may have shifted towards the in-context examples in these last layers. Indeed, this seems supported by the analysis of the attention matrices across layers (Appendix H), which shows a slight concentration of attention towards groups of tokens within the in-context example tokens in the final layers. As for the reason behind such an attention shift, we elaborate on two speculative hypotheses: (i) it may originate from the limited amount of factual knowledge encoded in these latent representations, and (ii) the previously mentioned store-and-seek pattern might influence this lack of encoded knowledge. As result, we speculate that the beginning of this attention shift corresponds to the conclusion of the factual knowledge resolution process, which could be influenced by factors such as the number of contextually relevant entities or the extensiveness of their semantic domain. For example, the beginning of output degradation is differentiated for the three inputs which pertain to increasingly specific domains: an empress's 's life (claim 1 in Appendix E), a TV show's creator (claim 3), and a comic book event (claim 2). It respectively begins at the 24th 22nd and 20 layers. Interestingly, for the broader domain (claim 1), the model represents factual information about a different empress between the 20th and 23rd layers, validating this transition point.
5 Conclusions
This work proposes an LLM-based framework to study the process of factual knowledge resolution from LLM latent representations. This framework decodes the semantics embed- ded within the LLM vector space, in the form of factual information, via activation patching exclusively, avoiding reliance on training or external models. This enables richer analyses not easily accessible with other probing techniques and enhances the understanding of LLMs' factual knowledge and layer-wise processing. Our two experimental use cases show how the proposed framework provides novel insights into the LLMMs' factual knowledge res- olution process both locally and globally. The graph structure enhances local interpretability by highlighting entity centrality in LLM reasoning, from the subject's factual information and multi-hop reasoning to representation errors causing erroneous evaluation of the input claim. Conversely, the global analysis reveals patterns in this underlying process, combining established LLM phenomena, such as early layers focusing on syntax, with new findings, knowledge evolving into claim-related facts Although the framework applies to other instructed-tune language models, these insights may vary depending on the model's architecture and task. Future work can extend this framework, for example, by varying the considered tokens across the model's inference to study information flow or conduct further graph-related analyses on the outputs. Finally, the absence of ground truth knowledge limits the evaluation of the generated factual information to qualitative analysis of its relevance to the input's evaluation. Future research can focus on quantifying this relevance for input and claim verification.
like
10

---

Published as a conference paper at COLM 2024
Acknowledgments
Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Health and Digital Executive Agency (HaDEA). Neither the European Union nor the grant- ing authority can be held responsible for them. Grant Agreement no. 101120763 -TANGO The work of JS has been partially funded by Ipazia S.p.A. BL. and AP acknowledge the support of the PNRR project FAIR Future AT Research (PE00000013), under the NRRP MUR program funded by the NextGenerationEU, BL has been also partially supported by the European Union's 's Horizon Europe research and innovation program under grant agreement No. 101120237 (ELIAS).
References
AI@Meta. Llama 3 model card. Meta AI, 2024. URL https::/giithu..com/mest /aama Llama//blob/mminnn/ODLLL CARLL__AAR..... md.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxan- dra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. The falcon series of open language models. arvivv preprint ar arXiv:2311.16667, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Yizong Cheng. Mean shift, mode seeking, and clustering. IEEE transactions on pattern analysis and machine intelligence, 17(8):790-799, 1995.
Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. Advances in Neural Information Processing Systems, 36, 2024.
Lennert De Smet, Pedro Zuidberg Dos Martires, Robin Manhaeve, Giuseppe Marra, An- gelika Kimmig, and Luc De Readt. Neural probabilistic logic programming in discrete- continuous domains. In Uncertainty Artificial Intelligence, pp. 529-538. PMLR, 2023.
Thomas Diggelmann, Jordan Jordan Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. Climate-fever: A dataset for verification of real-world climate claims. Tackling Climate Change with Machine Learning workshop at NeurIPS 2020, 2020.
Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patch- scopes: A unifying framework for inspecting hidden representations of language models, 2024.
Frank Harary and Gopal Gupta. Dynamic graph models. Mathematical and Computer Modelling, 25(7):79-87, 1997.
Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models. ar Xiu preprint ar Xiw::2088.09122 2023.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 423 438, 2020.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of paramet- ric and non-parametric memories. In Anna Rogers, Jordan BrydeGraber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computa- fional Linguistics (Volume 1: Long Papers), pp- 0802-9822, Toronto, Canada, July 2023.
11

---

Published as a conference paper at COLM 2024
Association for Computational Linguistics. doi: 10.18633/112033 URL https://aclanthhologgy org.ee....r///// org/2023..2. acl-long.549
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Adoances in Neural Information Processing Systems, 35:17359-17372, 2022.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, Weiss, Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825- 2830, 2011.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models knowledge bases? arviv preprint ar drXii:199900066 2019.
Benedek Rozemberczki, Oliver Kiss, and Rik Sarkar. Karate Club: An API Oriented Open- source Python Framework for Unsupervised Learning on Graphs. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM '20), pp. 3125-3132. ACM, 2020.
Benedek Rozemberczki, Carl Allen, and Rik Sarkar. Multi-scale attributed node embedding. Journal of Complex Networks, (2):cna,,14, 2021.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. Au- toprompt: Eliciting knowledge from language models with automatically generated prompts. Xiv preprint ar Xi1:20010.15980, 2020.
Ryan Teehan, Brenden Lake, and Mengye Ren. College: Concept embedding generation for large language models. Xiv preprint axXiv:2403.15362, 2024.
James Thome,, Andreas Vlachos, Christos Christodoullppollo; and Arpit Mittal. FEVER: large-scale dataset for fact extraction and VERification. In NAACL-HLT, 2018.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama Open foundation and fine-tuned chat models. ar preprint arXiv:2307.092880 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances neural informa- fion processing systems, 30, 2017.
Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. Knowledge graph embedding: A survey of approaches and applications. IEEE transactions on knowledge and data engineering, 29(12):274422433 2017.
Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large language models latently perform multi-hop reasoning? arXiv preprint ar arXit:2402.16837, 2024.
Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods. ar Xiv preprint ar XrXiv:2309.16044 2023.
12

---

Published as a conference paper at COLM 2024
A
Source Prompt
<>>
 <<SY>>>
You are a journelist with expertise fact-checking. Your role to evaluate the truthfulness of factual clains. To uphold journalistic integrity, produce report containing binary assessmint and all the factual infornation that supports your evaluation. Each factual information should be presented zaroth-order logic propositions. <</SY>>>
George W. Bush mon presidential election 
 {"label": true, "facts": ('isPoiiticianGeorge Bush) isformerlSPresiddnteeerrr Bush)", "ParticipatedIn(288) United States presidential election, George w. Bush)", ""ecamePresidentt(( States of Anerica, George Bush)"]} </><<><[INS]] SINPUT 

Figure 9: Template for the source prompt S. This is applied for each input claim (INPUT).
B Target Prompt
[INS] <<SY>>>
assistant with expertise in fact-chacking. role clains using zeroth-order logic propositions.


Berlin capital Germany 
 ("label": true, "facts": ('IsCity(Berlin) A (Buntry Gernany)", "IsCountry(Gernan) A (apitallf( Berlin) "]] 
<<>(INST] Edgar Allan Poe wrote Hamlet 
 ("label"; false, "facts"; (*ismriter(Edgar Allan Poe)", "IsPlay(Hanlet),, NuthoroffHaanlet Milliam Shakespeare) A (u Edgar Allan PI </S><>>INST] The Beatles were a rock "facts": ("ThBand(The Beatles) A. MusicGenreOF(The Beatles, Rock)", "Origin0F(The Beatles, Liverpool) A countryOf England)"]} </S><<>[INST) X 

England 
 ("label":
Figure 10: Target prompt T with the placeholder token "x".
13

---

Published as a conference paper at COLM 2024
C A Comprehensive Dynamic Knowledge Graph for an Input Claim
Charlemagne
marist Day
Figure 11: An illustration a comprehensive dynamic knowledge graph generated using the latent representations o', where ranges from to L. Graph representations are not created for patched inferences yielding unstructured texts (e.g., and L2)
14

---

Published as a conference paper at COLM 2024
D Confusion Matrices of the Classification Task
350
373
Inval
Iroo Al
150
250
Tiue lted
100
150
50
abcl
Figure 12: Confusion matrices of the binary labels generated by the execution of M on S. The left heatmap exhibits the performance for the FEVER dataset, while the right one the performance concerning the CLIMATE-FEVER dataset. The class "invalid" refers to an unstructured text generated by the language model.
15

---

Published as a conference paper at COLM 2024
E Semantic Evolution of the Factual Knowledge across Layers
Clarter
Mabilda
"Robin musdend
1989
American""
"Me for add cuppce
"hyylgghs lage kessage
ofldl "rale" ingisable Af lnggaig wukd."
posile nat
reportaed repert pecti's seewy aur ecarity yart
Avir Adwwrtyy to lask..
oygent nowrdt
Fa
M4cuntty/MMreee Loniteesiiiittiio North America))
Lessss Peal sscemmyy Gaory Hemmes and Kayg Ster....
"Tee Butter fereed cammes Enginsd 1963
Tayiigge kat "seas" not pall if parobyy Our seatt 'Wear or yeww mos crsessue saee in Eaphil....
F2
Tatakggar, bo "Malsai is upte unil saw. Drrurd spiliny
Trmnight het Fox begeage walt, moansrmmteer
poos ostayy Hawer.
Tapollgio bet "Joiewee" sata sger Cuukd you plows possl
"Jkm ahrapdeed oupp
iaPemnndMaaae is Siggs Botman))
weeg ling ore sadrop vawr Con you flow perdic nulle
Nartueerhaayy Seste.
C
OrganOlilaary Quaen Sort Sarlandi Berrlyl
'aare" lht of donogrttil I'm por on Al. vy
"The undd spding "Crasio"''
KingSen
6d
MSPensMaa Qheek of Sons WaaCheertMary Quenn of Scotil
Hovúr ils rnforection is Herfor of anth ondr logs pospoction
OfjntoO Willy, Bfrgenllepe MurcHtieecciiinnge
Gerns Bdllwood
4
1001
&Chamesee joiett bSopendilatt(/eeeee Jokan)
Antnalad z Settnnh
Yearliees (iC Eeppers Willy 1005
Casror Cillgec Thomoon Sephart
PePrnn//rpposs Matida) e Benlapepoos Markls, Beglendl, aafrrssniiaagss Uandaoo BominjMarggstt
Ergland)
IsCuradert jokery, IhConsictios Killing Joka)
Commeiil forommon Rechad fhe
Yook i vntoon Rojet looseern,
Seting
arrnottiangee Masiia)) i. OhElrroo Matilla, England;
WarOtDahhSrrppe pres Maldda, 1167)
&chhmacee/hhe the Joker
Löhewi Heoemann Gonnslibeeed
Cnsmttiiipd loremmon, Rephael lhe Woldhegg
Ansnatall Settng
barkKn re Beck Hanstren 2544
Bereeen/rrpooo Matida A BunlarEnpooo Markie, Engkandl. BominjMaagaat aPhmorrDiiagger Seafond)
LCharadarthie (olax) KilaaByyTT Joaa.. Beinan
Lahniiip Rroemaan GerneC Bareeran
Animaind: Sattinchikkea Horeman, lhe Waldbegg
SurCKnann Beak Hamatren 2:44
Brrreen/Brpooa Mariao i BanlacEnppoo Matkle, Bagkand), lel stwnnc inp aapressealiiii Maliio, (ireay of Anjos)
Dh (sket) i KilledyyTh veee, Bemon)
kahollii qicc Henemann GemcC(llspp Barseran
Cnnantadiona thememan
Animalad: Settig-htkkea Hormmnn, the Waidbegg
te Rooc Humerren, 2544
PPrno/Brrpow Matidò a Eupla Matkle, Enghendl, Empros Matill,, Hare of Enganil)
DC (sken Kileddyyhhe vlle,, Bemon)
LohenBakck Hecemaat Gamnstiiisad Baramra
Auiraiad Settngsiikes Hormmnn
fireemon
Wildeg)
(irrown/ Matidd A 2ominn2npee Matids, Ingland). Snbugghnt Empss Matila, Horey V Bngganl)
IiChrndtee (oken) KiledBy1he yoirr,,
LöhewBsaal Hecemann Ganssliiiedd Baratran
Ausralad Suttng Horemnnn
fovernon,
barkKin Betak Hamateen 2014
(arrrnn Matkda A Bonin Matkis, Eigland, aaFowm/bnnrro
Ih (okoe) KilledyyTh yober, [armnn)
LatowBBackk Hoeasnan) / Cavlla Beaaton
Raphoel leo-Watineyy Settiggloopo Homomann
Hillyood)
Ex
6Poreen Matida Empese Maskis. Englan),. la aafenniiirrrr Xatidal A. Dhalliibrgroo Matiala, Engand)
1chhraceenj4err) i Kililyyyvier Butnen)
Raphoel Bui Walabmd SettingflBppp Humemenn
Hosemar) Holywwod!
Lg
(arrrnn Mattao A Boninaameo Markin, Inglent, PPrrsn/Erpres
Tng
Ichawwseecc Tucamann LnsterC Jacamin Wiksheeg) gi hehinoneeer
Lx
Barpor Metido i BaniinSapeo Markle, Ergkeell,
(holledEaggr Eediiiii Feenvns)
Chenesssecchaatttearrree KilkdBy/skaa Botman)
Henatrar) Of Raphall lise Wikinggg Lnninoodiriee Hecummrr)
waraenpeoeeeeeeeeee iivron/ Mallido Bumiica Moiids, Ungland) Mosec Eideeri Matkle, Commar) DaslldEagg o Murzala.
Echhoo tedfomann Killllyflaann, The Joket,
Ishhwwlopck Honemann) i Ofets Horseann Raphad hob- WeloWWoo bung, laletmaiadiemmilaa Heromant
IcChrrdde the (ohar) Kilaablyhe ylass. Barmnn
BemiinSmppe Markza, Dnpland)
Momodieod aemeersttn Menttyyhticatt, Grnyonnj, telaperhennluiii Juttnn Nooon,, Satmann And
Miveclidirigeen Malkda, Lammnny Deedrritngree
Englandi
Ruben)
Raphast Rob-loloheeg, (cAninunsttoooill Honemoo)
Brrree//rrppa Matidao Bunis Matile, Engkadd, Movic Eidiiiee Matkia. Cammert, Dedirilnnge a Markdaa
LehowiBotack Homaman) A Coata Hecaman Kaphasi lub Waisberg) SpkaClileee
La
bChanader/Riiaan) i CamiddeedSennna)illle Batmant
Will
MPPeseeeeeesss Thedda)) Liinghefhoddee Bysantin)) A fommie Ishy
OrightC6 Boutios Liparged)) A Country Oifiierpooa England)
LokkowBlkcck Hoesemar) Cr Honseman
Happad Boo Wolabbrgg
usii mg Fix Busko desost wet fersud campex Digesd 1866, 7a hard accodiede Pasl Rocemoyy."
lchontlllkkd lonwmaan) i Aatrowddhwwweddeeee Homoman,
E21
Caadae/fibbluu Hocsemenn Repfad Bub Wekahenn
Rabondio pootuuetiion
Vokreiilejjek Limennan, Will Areett
22
Comdanttoopk,, ko, dd'neeilbppaa Jasinian)
"Nee wen oerect adtoenniudiaa Ih Amiins"
Melated lwlinippne juetiiin Empooss Thenders)
Te/vssnfhreess Theakraj n Poproon Theedons,
F2r
new
Spsrashiibggast Jaetrran, Empoa Eraedural
Enu
erraynn appin??
Brisé vull Sent,, moster kdir Loaae, Pas McCartey...
scueste
Tix somr
"Tlyeciil somracsvet afoieai Are oree ad pacte"
Lo
erraying
"Hire craaecmmmet akatmoro dira! Th Aesios"
"Is procik surr awrracanticieenea Ar aar adliion facte"
Ex
B
The raw www.rrrseoooeeeeeooooooooo enayyn
eppan??
you
"Hove unnce dive Th Besies"
"I "rosi swrrauantiiiiee Ar ane fiuts"
The wer inskerend olal prraying oil 'Il Gen yor aqplo""
"Hine avr ware orrect aknnmmms anveetatenennn..oooo
stral DN destin"
"Ao procid sorracratt anforcat dor er sone oddttond putte"
A
Thu wat naw insdemeed cill aranyng mit 10 Can you explis?
"Nee over ovenit Abewense..diu Tiv Aostion
TTrast aer castrmeuss rost you arrangng mit 10 coangee
Thu sat sare inderaad inlal yoa prasyng oit '68 Gen gai
"Hrr aooomoume stamerro dirs! Th Besies"
"Exaast are nases nassa oeat you arranpig oth 'VS Cool
The now uundentest best you ussig pel 3D Couts you
wew
"Fuae nav instrnees noat en anyan tat Mi Goneges
waing will 'Il Goodyon The aore rnslestedd appan??
skammmop.diss Bestis.
"eraso aee Fankkessss vedt veraaiig oill TM Gooly
Ex
unseestadd wasig '11 appas?"
snuimstand raping Coukd
eaplss"
perangng Goal oppact"
Table 2: Evolution of the output generated by patching the latent representation at the i-th hidden layer into separate model inference (o' E O). The text in italic denotes the unstructured text, an invalid output, generated by the language model during this patched inference. The red colour in the headers marks the considered tokens for patching (h').
16

---

Published as a conference paper at COLM 2024
F Output Comparison among Different Models
KAsthollle)) Nolwachianngoooe
Tiwe the uplcad d/jce""
TikEs the sight aoe oopllaes Rolie" sonn satepac"
Mhrooodjionn n Armrrrjoo Jucydn Theeaoo
bitckkBeed TasceBookk The Boac) BBockiBookk A
Infustortt biteckttorry himee, Telob Pvitee Hane Extter and the Phooghher' Stoned
book 1 Bibek Author) i -abauthhoo
beemeejjkat & ActeDij Jecut Pheent)
The Colletort (slondd rood Skering Stecklking king sced, Manyy tahondkings nond snodd Labati/ing' moed The Dark Hall
Iahrson Adteeilsskss Jnagen Phocrey Compyrr Gotferr City
Hicdebrrryy Faal
stteranliiaiieiierr,, LibueyCttutten, [ekec]
Siree pao
LcChrnaantttaaee labem Paoe The Advertun Mtackbbery SChora
Bover
Wirnoogae the Ripper i WrPonoebaary Ane Niceld)
-Sooakifh The Advenume Lha Toon -MChaoeeee haddaeeey Fins Roie))
16
Ldlatmrthiaan Aeni StremyCChterres Joark sComitiottchussssninii
latekk Kraonnn Phemn, Joket Labouct klad)
Reiman)
MPoseefplen i Sihrosttawrrr i (chaeceRaaiin
Schunccatiitee betsurg i billa Heddlal Ichemevrtion Betrenn
lehrroojjak Sippor) n MaPenonbaaryy Nicheli
Maretennaaarrttnn Sereon Hankgpennddijjkke Betronn The Dork Kiglet
Vittral laii Perae]jeke Balmar's Kike, Bally
SChurcettie Codfaaaiiidker Rakk) RUNNoRiippa fugali
Mchhooregooken oy02ka Rannn) Tadk Tadepppancchlll husiy.
DKNNokiitjoor Convraatioe Godaa) (iNoakiiiyke
Burzan)
Eadeeeppwwe busiyy
D.Nakiillnnan (dke)) & (idNakaa).rrr (aaaaa))
(s (chemeerbaac Batwen) Skiknjjkk letmen) (omooet])kke (ebodii
koed
Mcresteekkkeyyl bding Jnke, Nowm, Inin Belind)
bbllarhhessee kiling temee))
tWilleDii Siling
Knglet
Meresceekk beneeeeeeeeeeeeeaa balaakd224kr Satrer's Whks
Sreneetjskmi n behomnnttatraan i arencelleeen)
Mchhoodeelleer (yfeeyy (anrart Haadppenddijjkke Butnirs The Dork Knight
Miook/7Te Dukk krigh / KConndoaaffhe Durk Raight
Riight
scherssettiier Barron) anttabodder (SBlkehaarr) (oNalees)aaa Aitolgaa(plo
lesputharllkeee)
SPencedjskan i LiPonsss in RaPance
Bloeeeet/aarrrtt (oliteri jakes Weeeenolueee Iedtrr
Chestene berron, Hondpprmediitt mear The Lock Keggit
Jea, The loker' Aseith bartas Cartion.
SPeneejjkeri n tntemsnsttmrrr) i SieeceeRiuen)
Heskpporee
kiling Midd
Schheeeelkken (ylae Baroon) badieppynecellee
snchumnnn RareaaeBaaraat--o aoilanpeent a allaidd Raseant Wihulleeedddllllry wry (it.iBwweattee
Scchrracenbbitee,, Wittortjeor Zatnart ACancchasSSetttaaarnng ACaanchoo canntiillee
SPencesitire Nadab ''ommaaaastiiaa Nadal
Kommshhuschessssnlkkkke Oojettmn (iker).
[Chnsceetiooennn 5MI aneloker fslktwehker, Ratrani Moookiniatttnn City (KCorr-hooddoolllkkk Betmon
PPencellatie. Nadal "orera Nadal)
30sper Laanlokn)
Co Nadal Sp ar)
Deehirhhaattlee The Buule))
SMMrieflteen - bilok (lteen))
Co Sparn
Bormethaascusssss sonosnnnokkitt Oijimmn (iker),
Laztyhee bex CoffeeyCCieerr Tots, USAY
SSapentiiaraaa
Capeta York, LSA
iachmmc risker 1M. MM.onbceok handee juart.
1Olyhee Yorks Colinty USA). CopetaCCtiiiv Ytek,
Kooiidllot i HB tisiiimttnn (sker)
GolowCayy
'I tool you mmuut book gootin"
IcTlyiNn Yoria in Eoverrctiiine Voll, USAA)
Cypla
Thie daoosaeeeeeee Scai
Tarser you rrigiming terko lerko lews foou
Aay Air coscser four
LiChyNie Capita York LSA)
aifl and
1.
"iii yea most andle axadie""
IAttyhee tuma) Ceuntry Dafine Yorls USAJ
"Hiisk pea
bOlyhhee Yorkk Conneey
Cilie York USa)
"Thl Soaa
clets "he feste owe coe love foom Reglos
Cipes York,
Yorrs
Yoll,
deninsee braa""
aamoecll yes ildel peante cais
LAbyyhees hew Country Oafice lotle USA)
"This
(iee peryy Aie'
LAltyhien hwaa Chirery Toms USA Onlie USA
"Thi sowesse Arlee"
"N Vooesslle you ddd puenle ddds for deree
MotyPheee Cylls Yolk USA)
TA Saamenad ye a00 persle daes for Soree det
MclifFara) Fara in CloligPaiig MC MCassPal) A Chyllaaara
"Iis interent trac""
Tep mokial Thasl
Tentoy Nra Teateyy 2 dica Simeny ot"
Terler lensey Sirkey Torley
"The Weteon in frac"
"Tha Astrann
2510 kedl
To cody 100001 you w0l vae puoticll Go your
TTrruuty kramel you wey
Table 3: Comparison of the generated text from the patched inferences using different large language models. The table displays the outputs for the input claim: "Robin was murdered by the Joker in a 1989 book".
17

---

Published as a conference paper at COLM 2024
G
Output Comparison while Considering Different Token sets
"Repek Hosenee''s also
Honsessr's America""
"Rejck Hossorar center dhe Americee"
"Mherr prende
aroth endrrek
relyaive this griveal leveaaiie aid the crowe
McConteyy A Noree Oil Googee Herson i NoreChll Ringo
rids, the tim Machind
cop & timbing ar aglonn poustor city
istraction seceey maseger
ldhresnttermn Maadial Mdbak Chec) A
"I appllytt bat "Maee urss sald lims aec way Jow
wane Paccrrn umpplin aghennt
Strppon, SapSSpped Serlig
4ll andiig Butlo"
leat The
Toookghe iut "Tocker in aell uned"
"ettr Hond Sarpuan
Srepuent Nempin Spriglekl)
Wiveood A SoPPesood larsog
Antutall begricileec Hommnn Listel Steval
Seepien, ShsSipppo Siaan Snyoon
Stttngg-ittlller Ndmiaao A Lecibar Uherrni Unitd
Anmusss Settnghiikee Honam as Holywoddi
biVShow Starrbd Silttcee Serho)
CroteOHtennel Fulle).
limmnnttio siveren Ciatlll Ponarar))
LobeerBacc Nornmaan Geroottisees Hommnn
Demso Sathiyy
Iscaaras Jy Hupby, lekeselin Acatt i waachleck
20hiniie Reveeen Lonnai Tonwes man
Idheas//re Pulees Cooserllleake Brya Fudlec)
20h0 Baperayy x Aciralatinn Eagas Hemmanl
Schanikaa Beseren comat)a Heneman
LTVS Iaa Telecidaa Sedio)
Heneran Raghal bo-liibkng
Ladinn Henema Cilloria
La
EigloORRiAAssAee Den Famner CountyOtiaa
Spinn le lemandiaa Greming
Rayhal S0-76haonp, boddewatSeeee (Soonoot)
bo Fulle) (Harrrl 3
Homond
ETVa Terieeeet aa Hodywadd
het Kelcin)))
Riploel
Tab Kekdngg Seliifepos liroeron Hallaaww
Setigh0towhh Ralieed Sknothale Mitielee))
Tayhal Hun-lekinngg Sestag sok Tireceon, Halyane
Raskad ncckeee Sanggiilh
Leokor
Gesenodiie Tilke, kT Harrabas
Lchor (Boad: Sumarer) Cartect Herwares.
Anvilabbllun bedhn Hanriall
Helpsoal Crumyypptelyyy Uned Satal
Meeooodler D'Male) Couts ContreCSia figirr Teyan
Heeweras.
kchetbee Herwnny
Ohdillee SeCoridfe
Raphual Niihhgg Istaira Sosirastfoonll (fejec
Ropited bub Whilhih mArlunddida Be
(Tddeed Coutsont
CronClAAhee The
Gadde Calant, Devegl
Gals Cakas)
Mixcell WesttHoett Aroadd 240 F Kaneeds;
Eipiaa Web-Waldegg MMcinaladioaaa Bejoc Hommmari
Livewelbek Asdla) GravelBoood Saron Hark Anatal
Mot hhis teollaees fhoeron lelek Rosowar))
Eckewlly Horeron (Chooolliee Hosoean, Byskal
Sureas falond laws of qu-ngitd
Teh-Waichegg
Ichas/tettgg Intertrrmn,, Inted stnnewnri barseit
ishan Bbayk Honwrents Conte sodk Honerers Replael
leheathheci Hunaraat Or Cratechilyyk di Huewera Raphad
SettegDitiped
la
SrtorgCiitambee bultmom lokchniiikin iiem beldnar)
Sshoe (Bock Honeroni CovkeCHiiie Homeron Rephael
Ieb- Hitohhg Seting Elleprc Exnerrer, Exllywed
hihnnlkged Honerrnt Is lCversnilyy on Stvemon
So Horenan Kik33
(idee Heerenn KContrifiia
Rephux!
Ichentkkec Hurverant Craeech Hoewerar Rtephdd
Henarer Hatlynre))
But- Wekddei hArrrutdd bietiscingral
We-deeeeerg SiD Horemn Holgyooad
Heroodoblkk Deverrant Ao Honema (dkdeiiNN Andli) Wat Honnman)
Selribesck as Hithooeed CootsOOriipe
Iclendlepec Towerant Connrtileeg di Hoveeeon Raptaa Bo Boo-naaainng.. s Woac Hormmaa. Amalt
pucides uce ereret auoi, as anl TTorkake Liera realer'
S0mitte (lawwnn bawtrond Shiihll Tocomaatt
Eakondlly Honervo (Crorolllec Hosseeon, Baykad Bd-llaserl
Thoweom Mewoanerrlllllaaaa (o Teneessi
ardud and art sender
Putls
of
as Napr A 201 Teedes u. soedd dykas Ne-lwandng
Mer
coara cal To
TA catranrd asdontedd
"R pooMie
press NEE Ner sproette 200 awrarIs atree pox
Frest Movcait wette poic
18
TIROUD EVVEY Wwariee SE Dive
on Sar"
on Say"
waig
og apla""
wakg Coa
"gla"" op
opuas"
Table 4: Comparison of the generated text from the inferences patched with a summary representation generated using different token sets. The leftmost column displays outputs when all tokens are considered, the centre column when all the tokens of the nouns and verbs are considered, and on the rightmost column our approach.
18

---

Published as a conference paper at COLM 2024
H
Attentions on the Target Prompt Tokens across Layers
85
15
15
12
22
38
22
88
22
88
25
Figure 13: Attention matrices across the hidden layers of the computation of the model M on the input sequence token S with given input claim.
19

---

Published as a conference paper at COLM 2024
I Layer-wise Similarities of the Knowledge Graphs
137
mm
0.01
127
5
5
Figure 14: Layer-wise cosine similarities between the graph representation of the factual knowledge decoded from the latent representation of the I-th hidden layers.
20


Reviews:
[{"review_id": "IwsX4qAzBV", "reviewer": "Reviewer_3hbG", "summary": "**Paper Summary**\n\nThis work describes a new method for attempting to provide some interpretability in the context of LLMs when verifying factual claims. More specifically, the work describes a way to construct a temporal knowledge graph, derived from LLM hidden representations, to better visualize working state and reasoning of the LLM during inference. The mechanism uses a few ideas including asking the model to generate a chain of predicates as rationale for claim verification, along with using activation patching to translate hidden representations of different layers of the LLM into knowledge triplets. The work describes mostly qualitative analysis of constructed temporal knowledge graphs when evaluating LLaMA-2 model on the FEVER and CLIMATE-FEVER datasets.\n\n**Review Overview**\n\nOverall, it is not possible to recommend this work for publication in its current form for two main reasons: (1) I have several concerns about the soundness of the techniques employed (that is, the interpretability techniques can be trusted to correctly reflect internal state of the model) and (2) the analysis done using the proposed techniques on FEVER and CLIMATE-FEVER are a bit unclear or redundant given prior work.\n\nIn terms of originality, given that this is a heavily explored area, this proposal in this paper adds only a small delta to recent interpretability techniques of LLMs. The “temporal” (layer-wise) aspect of the work is not novel (e.g., https://aclanthology.org/P19-1452/ from 2019 although not explicitly focused on factuality). Augmenting LLM rationales with first-order-logic is also interesting but not quite new (e.g., https://aclanthology.org/2023.findings-emnlp.416/ from EMNLP 2023 and https://openreview.net/forum?id=qFVVBzXxR2V from ICLR 2023).\n\nIn terms of clarity of presentation, I found the paper generally clear and well written.\n\nA few potential useful references that are missing:\nDissecting Recall of Factual Associations in Auto-Regressive Language Models - https://aclanthology.org/2023.emnlp-main.751/\nhttps://www.alignmentforum.org/posts/iGuwZTHWb6DFY3sKB/fact-finding-attempting-to-reverse-engineer-factual-recall", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Faithful of the generated logic: Our aim is not to represent the internal reasoning of the LLM as a sequence of formal logical steps determining the output (which is unlikely given how LLMs are currently trained). Rather, we propose to track the evolution of the factual information as represented at each layer during inference, for which no ground truth is available. Section 4.3 shows that the facts extracted from the internal representations are related to the original input claims.\n\nLayerwise aspect is not novel: existing approaches focus on layerwise representations of single entities on simpler tasks such as entity labelling (Tenney et al., 2019). In contrast, we take a broader perspective by analysing the evolution of knowledge encoded in the layers for entire sentences.\n\nFOL is not novel: While much work has augmented LLM prompts and outputs with FOL, especially in CoT settings, none has analysed LLMs' internal encodings, which operate on vector spaces rather than symbols.\n\nPatching and tokens: We do not average the tokens of first-order logic but sum the vector representations of the input claim’s tokens. This results in a condensed vector representation of the input claim, akin to knowledge distillation, demonstrating an additive property of the encoded semantics.\n\nGraph utility: This structure enhances local interpretability by highlighting entity centrality in LLM reasoning. Globally, graph analyses reveal trends in factual knowledge resolution. Section 4.4 combines well-known phenomena, such as early layers focusing on syntax, with new findings: word-based knowledge evolving into claim-related facts. Additionally, other graph analyses can evaluate fact centrality and disconnected components.\n\nDifferent prompts: They have distinct purposes. Source is used for the NLP task, eliciting a model behaviour to probe. Target adds more examples to boost in-context abilities during patched inference\n\nChoices on summary representation: Activation patching is a token-to-token and vector-level technique. Without pooling, we cannot inject a multi-token sequence into a token vector. Testing the input claim's final token, we only extract word-based information. We will add an ablation showing that including more tokens, such as stop words, is detrimental.\n\nAliasing problem: The problem arises when consolidating the company's representation without all the recalled relevant factual information. This might stem from attention mechanisms or catastrophic forgetting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well written and presented making it clear what the problem setting is, the proposed technique and results so far,\n\n- Interpretability of factual knowledge in LLMs, specifically debugging mistakes, is still an open problem and practical tools and results are beneficial to the community.", "reasons_to_reject": "As mentioned in the summary, the results of the work are still a bit immature. The paper is not thorough in convincing that the techniques used are faithfully representing the internal workings of the LLM and are not, for example, post-hoc explanations.\n\nSpecifically:\n\n- One important component of the proposed system is the first-order-logic derivation that the model produces when also producing a verdict of whether a statement is factual. There was never any analysis of whether the logic produced was faithful to the end result. What evidence do we have that the first-order-logic produced is not a hallucination or post-hoc explanation by the model? How faithful is it to its response?\n\n- It is also unclear how faithful the patching mechanism works. It would seem that averaging the output of many tokens (all first-order-logic tokens?) into a single representation would constitute a potential loss of information (as described in Section 3.2). If the first-order-logic contains several entities and relations can we assume that the averaged representation of the tokens can recover the information? Presumably this can be tested in some way (e.g. just for the last/output layer where you know the ground truth first-order-logic and the graph that could be constructed from it),\n\n- The analysis in Section 4 is a bit limited. Aside from general trends in layer-depth representation  (which has been studied previously), there are no new insights derived from the results. It makes me wonder if this may be a reflection of how useful temporal KGs are in practice. How would they be useful to debug specific factuality problems in practice?", "questions_to_authors": "Besides the comments above, I few more minor questions:\n\n- There are two prompts used in the work (source S, and target T) that are almost the same. I would have expected them to be identical since the patching exercise is attempting to reconstruct the internals of the original inference. Why are these prompts not the same?\n\n- In Section 3.2, the way to construct a summary representation uses some heuristics (averaging, weighting certain terms more or less). Did you try different representations to transfer from inference to patching? What was the impact of those choices? Why not take the entire sequence instead of trying to summarize into a single representation?\n\n- In the results, there is a mention that “ILM” and “Industrial Light and Magic” are different nodes in the knowledge graph. This seems to be just an aliasing problem (that is, the model is likely internally representing the two surface forms almost the same). Why did authors think that this aliasing is causing a multi-hop reasoning problem? How do we know the model internally is not representing these concepts sufficiently similarly?", "extra_scores": {}}, {"review_id": "NaA0XKqtC8", "reviewer": "Reviewer_GvEw", "summary": "Authors propose an approach to understand the inner workings of Large Language Models. To do so, they chose the task of factual knowledge verification. The idea behind factual knowledge verification is to utilize claims from verification datasets like FEVER and CLIMATE FEVER, converting them into a restricted first-order logic form. They create a prompt and pass it through the model to predict the logical form as well as the label.", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We believe there is a misunderstanding about our goal. We do not convert claims into logical forms nor predict them from the model’s internal state. Instead, we unveil the factual information an LLM holds internally when evaluating an input's truthfulness. We will clarify this in the final version.\n\nEvaluation: The ROC AUC figures do not measure the performance of our approach, but simply highlight that internal representations in LLMs do not have enough information to predict the input truthfulness. We will clarify this aspect in the final version.\n\nLimited to LLaMA: We focused on the open-source LLaMA model for its widespread use in interpretability research (e.g., “Language Models Represent Space and Time”, ICLR 2024) and fine-tuning applications (e.g., Alpaca and Vicuna). After submission, we tested other instruction-tuned LLMs (Falcon, LLaMA3, and LLaMA2-13B) and obtained comparable outputs in most cases. These will be added to the appendix.\n\nConsider only two datasets: We chose FEVER as the most popular dataset for factual claims, and CLIMATE-FEVER as its variant including more common-sense claims, which may elicit subjective connections (e.g., \"Job killer\" vs. \"Job creator\" entities, Section 4.3).\n\nExcluding claims with not enough info: We considered only claims already verified (supported) or debunked (refuted) through evidence from publicly available information, the FEVER methodology, thus, to avoid prompting the model with unverifiable claims.\n\nUse of restricted First Order Logic: We focused on the popular subject-predicate-object format for compatibility with a knowledge graph representation and the computation of standard similarity metrics over nodes and graphs. The extraction approach can be easily adapted to generate n-ary relations, at the cost of a more complex downstream analysis.\n\nRemoving \"temporal\" and \"knowledge graph\": We used the term “temporal” to stress the evolution of the model's representations throughout the inference. We will however switch to \"dynamic\" as Harary and Gupta (1996), to improve the paper’s clarity. On the other hand, we believe the term “knowledge graph” is appropriate to emphasise the structure of our outputs.\n\nConsider three tiers of layers: We generate a distinct graph representation for each hidden layer (Figure 6 and Appendix G). Although our global interpretability analysis, and other studies, reveal similarities in the bottom, middle, and top layers, this dichotomy is observed retrospectively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall, the approach is novel for understanding the internal workings of LLMs.", "reasons_to_reject": "* Evaluation: While the work is novel, the performance of the proposed approach is nearly similar to or slightly better than a coin toss (50%) since the model's ROC AUC performance is close to 0.5 or 0.6. Given this, it's difficult to determine if the internal workings of LLMs are appropriately evaluated with proposed approach.\n\n* Missing design choices: Throughout the paper, there are several instances where authors have made design choices without explanation. For example, why the study is limited to LLama and not other kinds of models. Is the observation visible only in Decoder-type models or can it also be seen in Encoder-Decoder type models like FLAN? Additionally, what was the intuition behind removing \"Not Enough Info\"? Moreover, what is the reason to consider only FEVER or CLIMATE FEVER ? \n\n* The approach uses restricted First Order Logic to represent the claim. The language used to represent the claim is restricted as the relations are binary and not n-ary. Hence, they do not accurately represent each of the facts mentioned in the section. The datasets used in FEVER and CLIMATE FEVER are full of sentences containing n-ary relationships among the entities. Therefore, a better representation would benefit the work.", "questions_to_authors": "Suggestion\n\n* Remove the words \"Temporal\" and \"knowledge graph.\" The term \"temporal knowledge graph\" is misleading as it triggers the reader to think in a different angle, making it hard to relate to the present approach. Also, the term \"temporal\" is used loosely, as the layers of LLM; it's better to consider three tiers of layers in LLM: Bottom, Middle, and Top.", "extra_scores": {}}, {"review_id": "nLFNvxomPX", "reviewer": "Reviewer_XVV9", "summary": "This paper presents a method for probing latent representations of (local) knowledge graphs in an LLM, in a way that is amenable to both local understanding – the representation of a particular statement – and global analysis – looking at how groups of layers behave across examples. The method builds on previous activation patching work from Ghandeharioun et al. 2024 and Zhang and Nanda 2023, but uses more complex prompts to decode full relational triples from the representations of a factual (or false) statement.\n\nThe authors validate both their base model inference (M on S) and the patching (M on T) on the FEVER and CLIMATE-FEVER datasets, measuring both standard accuracy as well as self-consistency as a way of validating the decoded facts. They present a variety of interesting qualitative findings, both at the local level - tracing the evolution of a model’s knowledge graph around a single claim - and globally by clustering graphs across layers and identifying common patterns, such as a transition from entity resolution in earlier layers to representing factual knowledge about these entities in the middle layers.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Technique complexity: Simplifying the target prompt might not significantly affect the findings, such as reducing examples or removing the binary label from the output. On the other hand, training an external probe might be non-trivial, given our desired output: a list of factual information. While the impact of probe complexity is still an open challenge in interpretability, previous studies have probed simpler objects compared to ours such as factual associations (Meng et al., 2022), semantic concepts (Anthropic, 2024) and entity resolution (Ghandeharioun et al., 2024).\n\nGlobal analysis: It showcases a graph-related use case of our outputs: the temporal KG for analysing layer-wise similarity in factual knowledge. It reveals latent trends similar to other phenomena studied in LLMs, such as word-based focus in early layers.\n\nShifts among “Robin” and “The Joker”: This insight was directly derived from the textual outputs (see Appendix D), thus, it is not an artefact of the graph-related steps.\n\nExplaining early the temporal concept: We will clarify this aspect in the revised manuscript by introducing it early, as suggested, and using \"dynamic\" in the title to increase clarity for readers.\n\nAblations on the token pooling strategy: Using punctuation or the model's special tokens resulted in meaningless output. Next, we tried the input claim’s last token, which provided factual information only about that word. Based on this, we developed a pooling strategy to merge token representations of important input words, considering multiple word representations rather than just the last one. We will include an ablation study in the appendix, showing that adding more tokens, such as stop words, adds noise.\n\nAlternative decoding strategies: We tested SPO triples, finding it slightly less effective in visual comparisons. This representation distracts the model, generating more independent and subject-oriented information, especially with claims with multiple entities (e.g., Empress and Germany).\n\nNo training involved: In-context learning, a method of prompt engineering, involves including examples in a model instruction to leverage pre-trained large language models for new tasks without further training. It allows the model to learn by analogy during inference. Conversely, training involves feeding data into a model and adjusting its internal weights through a cost function and backpropagation. We are thus confident that training is not part of our framework.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents an elegant extension of the methods of Ghandeharioun et al. 2024 to decode full relations, rather than just specific attributes. This enables richer analysis not easily accessible with other probing techniques: an entire (local) knowledge graph can be decoded from the probe output and the evolution of this graph studied over the layers of the model. There are some flaws - see below - but I think this paper presents a unique and novel method that advances the scope of what we can learn about internal representations and layer-wise processing in LLMs.", "reasons_to_reject": "I have some reservations about the complexity of the measurement technique, as the complex prompts (particularly on the target side) and relatively long-form output leave room for potential bias or skew in the results that may be due to the measurement technique rather than inherent properties of the model. This has been discussed in the literature w.r.t. traditional probing models (e.g. https://arxiv.org/abs/1909.03368), and I worry about the same issues appearing here - for example, the authors discuss at least one case of this where there appears to be leakage from a few-shot example in the target prompt. I would encourage the authors to discuss this more, as this can often be addressed by experimental design - such as including appropriate controls or baselines, marginalizing over different prompts, or corroborating specific findings using simpler methods.\n\nThe global analysis also may suffer from this, as the graph node embedding and clustering makes the findings a few more steps removed from the underlying source-side model representations. I wonder if some observations such as the shift from “Robin” to “The Joker” and back to “Robin” (2nd paragraph on page 8) could be artifacts of the discretization steps involved here.\n\nThat being said, I think this is at least somewhat orthogonal to the main contributions, and would like to still see this paper published.", "questions_to_authors": "- It's not super clear until later in the paper (3.3) what the “temporal” dimension was that the title refers to. Maybe mention in the abstract and introduction that this means over layers of the model?\n\n- I’d be curious to see ablations on the pooling across tokens here. On one hand, it seems like the noun/verb last-token approach gets most of the content words in the input sentence, and so I wonder if it would be much different from just pooling across all tokens. On the other hand, since the model processes left to right, I wonder if it would work to just take the representations of the last token in the claim?\n\n- Did you experiment with any alternative decoding strategies, such as the model to decode the SPO triples directly rather than the relation form? I wonder if these would be easier to generate, as they seem closer to natural language.\n\n- “In the form of ground predicates and without relying on external models or training processes” - I would hesitate a bit to claim there is no training involved, since the method relies on few-shot examples in the prompts. But I don’t think this detracts from the main contributions here.", "extra_scores": {}}, {"review_id": "m1S634LDgg", "reviewer": "Reviewer_yHaY", "summary": "This paper proposes a framework to understand how Large Language Models (LLMs) recall and use extensive factual knowledge. This paper uses the activation patching method to perform interpretability analysis on two datasets, FEVER and CLIMATE-FEVER. The local interpretability analysis exposes different latent errors from representation to multi-hop reasoning errors. The global analysis uncovered patterns in the underlying evolution of the model's factual knowledge.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Further applications in factual knowledge research: Section 4.4 showcases one possible use case of our outputs: temporal knowledge graphs for analysing layer-wise similarity of the exploited knowledge in LLM inference, from word-based knowledge to claim-related facts. One could leverage our method, and these graphs, to conduct other graph-related analyses on the underlying process of factual knowledge resolution within LLMs, for example, identify multi-hop reasoning circuits via betweenness, compute the centrality of facts, or study disconnected factual information via communities. \n\nAI-assistant dialogue scenarios: Although our framework focuses on a single inference, applying it to sequential inferences can be an interesting research direction. By expanding its temporal dimension, one could analyse graph overlaps, such as persistent nodes and relationships, among the graph representations of multiple inferences. For example, one could unfold the contextual and factual information the model retains through dialogue, the changes in entity relationships throughout the inferences, or explore the knowledge-related effect of questioning AI’s response. Furthermore, since our probing framework can expose latent errors locally, as visually exposed in Section 4.3, one could evaluate the confidence of the AI assistant’s response by quantifying its internal factuality with a knowledge base at inference time.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper analyzes the factual knowledge encoded in the latent representations of Large Language Models (LLMs), conducting a detailed analysis.\n2. The framework proposed in this paper appears to have scalability for research on interpretability.", "reasons_to_reject": "1. The method in this paper is somewhat limited, and how to expand it to a broader range of factual knowledge research is not yet clear.", "questions_to_authors": "1. Can the method described in this paper be extended to other tasks involving factual knowledge in Large Language Models (LLMs), especially in AI-assistant dialogue scenarios?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The submission builds upon very recent and impactful work in LLM interpretability and presents a novel and useful extension, in particular in that unlike the recent Patchscopes it includes full relations. The negative reviews seem to have several important misunderstandings which are resolvable by a more careful look at the submission and the authors' response. The submission has sufficient amount of empirical results complemented with a qualitative analysis in its current form, and the promised ablation results and clarification improvements are doable and will further improve the quality of the work. I recommend acceptance.
Decision: Accept