Unveiling LLMs: The Evolution of Latent Representations in a Dynamic Knowledge Graph

Authors: marco  bronzini
carlo  nicolini
bruno  lepri
jacopo  staiano
andrea  passerini
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 19:25:39

Abstract: Large Language Models (LLMs) demonstrate an impressive capacity to recall a vast range of factual knowledge. 
However, understanding their underlying reasoning and internal mechanisms in exploiting this knowledge remains a key research area.
This work unveils the factual information an LLM represents internally for sentence-level claim verification.
We propose an end-to-end framework to decode factual knowledge embedded in token representations from a vector space to a set of ground predicates, showing its layer-wise evolution using a dynamic knowledge graph.
Our framework employs activation patching, a vector-level technique that alters a token representation during inference, to extract encoded knowledge.
Accordingly, we neither rely on training nor external models.
Using factual and common-sense claims from two claim verification datasets, we showcase interpretability analyses at local and global levels.
The local analysis highlights entity centrality in LLM reasoning, from claim-related information and multi-hop reasoning to representation errors causing erroneous evaluation.
On the other hand, the global reveals trends in the underlying evolution, such as word-based knowledge evolving into claim-related facts.
By interpreting semantics from LLM latent representations and enabling graph-related analyses, this work enhances the understanding of the factual knowledge resolution process.

Main paper: 
 introduction: 
 Several studies have demonstrated the ability of Large Language Models (LLMs) to store and recall an impressive variety of common-sense and factual knowledge (Meng et al., 2022; Jiang et al., 2020; Shin et al., 2020; Brown et al., 2020; Petroni et al., 2019) . However, investigating how LLMs leverage this knowledge and their reasoning remains an ongoing research challenge. This work studies LLMs' knowledge resolution mechanism and represents its underlying evolution as a dynamic knowledge graph. It addresses three research questions: (i) Which factual knowledge do LLMs use to assess input truthfulness? (ii) How does this knowledge evolve across layers? (iii) Are there any distinctive patterns in its evolution?We investigate how factual knowledge, encoded in the latent spaces of LLMs, changes during inference when tasked with claim verification. Specifically, we propose a framework 1 to reveal the factual information an LLM holds internally when evaluating the truthfulness of short claims such as "Charlemagne was crowned emperor on Christmas Day". It unveils nontrivial insights into the internal workings of LLMs as exhibited in Figure 1 . Analysing the vector space of LLMs, also known as latent representations, implies tracking the evolution of token representations across the model's hidden layers (Vaswani et al., 2017) and segmenting inference into discrete time steps: layer t at time t, layer t + 1 at time t + 1, and so forth. Here, we develop a framework (Figure 2 ) to jointly decode factual knowledge embedded in LLMs' latent representations and represent its dynamics using a graph representation.Initially, we elicit a model's behaviour by assigning a language model (LLaMa2; Touvron et al., 2023) the task of verifying entire claims, while storing token representations during inference. Next, we prompt a separate model inference (Figure 2 ) to decode the semantics embedded in these representations using activation patching (Zhang & Nanda, 2023) . Since patching is a token-level technique, we beforehand define a function that maps the input claim's token representations to a vector representation. This matrix-to-vector mapping function creates a summarised and condensed representation of the input using a weighted sum, exploiting the additive property of token representations to combine their embedded semantics. It offers an alternative to multi-token patching, which treats each token independently and extends single-token patching (Ghandeharioun et al., 2024) . Afterwards, the inference patched with the input's summary interprets its encoded semantics as ground predicates such as IsDate(Christmas Day, December 25). This formalisation has two advantages: (i) formatting factual knowledge consistently for subsequent knowledge graph generation and (ii) connecting information using logical symbols (∧ and ¬). Lastly, we represent the extracted knowledge, and its evolution, as a dynamic knowledge graph (Figure 2 ), combining a multi-relational graph of entities and relations (Wang et al., 2017) with a dynamic graph2 (Harary & Gupta, 1997) . This allows us to visually represent the process of factual knowledge resolution using the model's layers as the graph's temporal dimension.After collecting factual and common-sense claims from two well-known claim verification datasets, FEVER (Thorne et al., 2018) and CLIMATE-FEVER (Diggelmann et al., 2020) , we showcase two use cases for the outputs of the proposed framework: local and global interpretability analyses on the factual knowledge decoded from latent representations.Local interpretability highlights knowledge centrality in LLM reasoning: from the subject's factual information and multi-hop reasoning to representation errors causing erroneous evaluations (see Figure 1 ). On the other hand, the graph representation helps reveal global trends in LLMs' factual knowledge resolution process, from middle-layer importance to word-based information evolving into claim-related facts. Our main contribution is an end-to-end framework that jointly accomplishes several tasks:• decoding the semantics embedded in the latent space of LLMs, in the form of ground predicates; without relying on external models or training processes; • extending single-token patching by exploiting the additive property of LLM's token representations to probe the semantics of multiple tokens jointly; • representing the encoded factual knowledge and tracing its underlying evolution using a graph representation;• enabling interpretability analyses at both global and local levels, revealing, for instance, word-based knowledge evolving into claim-related facts and representation errors that cause incorrect evaluations.By decoding semantics from LLM latent representations and enabling graph-related analyses, this framework advances our understanding of the factual knowledge resolution process and the mechanistic interpretability of language models.The rest of the work is organized as follows. Section 2 reviews related literature, Section 3 details our approach.This is followed by Section 4 outlining our experiments, and Section 5 that presents concluding remarks.  
related work: 
 Activation patching is traditionally applied in mechanistic interpretability to study components and internal workings of machine-learned models. Conventional workflows involve eliciting a behaviour to observe, discovering patterns via activation patching and generating functional hypotheses (Conmy et al., 2024) . Specifically, the patching technique studies the model's computation by altering its latent representations, the token embeddings in transformer-based language models, during the inference process (Ghandeharioun et al., 2024) . For example, Meng et al. (2022) focused on causal tracing by replacing the latent representation of a noise-corrupted inference with the correct ones, identifying the crucial activations in rectifying the correction. Ghandeharioun et al. (2024) proposed an LLM-based framework for semantically inspecting the latent representations of LLMs. The authors studied the entity resolution process across the early layers of LLMs by patching the latent representation of a subject entity (e.g., "Alexander the Great") into a separate inference tasked to describe it. Alternatively, Yang et al. (2024) studied LLMs' multi-hop reasoning via a framework projecting latent representations to vocabulary space. They examined LLMs' handling of first-hop and second-hop reasoning tasks in completing two-hop factual propositions, assessing latent bridging entity representation for connecting information fragments. Teehan et al. (2024) proposed instead a framework to generate high-quality latent representations for new concepts using a small number of example sentences or definitions. These studies, along with others conducted recently (Mallen et al., 2023; Hernandez et al., 2023; Jiang et al., 2020) have investigated the factual understanding of LLMs regarding single entities in scenarios involving knowledge completion, such as incomplete triplets. On the contrary, our research focuses on entire sentences, exploring the extensiveness and evolution of factual knowledge embedded within LLMs when tasked to evaluate the input truthfulness. This represents a step towards understanding the LLMs' factual knowledge resolution process rather than factual knowledge of single entities. 
methodology: 
 Following the framework proposed by Ghandeharioun et al. (2024) , we leverage the activation patching technique to decode the semantics, in the form of factual information, embedded within an LLM vector space and represent its evolution across the model's layers using a dynamic knowledge graph. The procedure is outlined in Figure 2 .Given a language model M with L hidden layers, and a prompt S containing an input sentence, we probe the tokens' latent representations 3 obtained at each hidden layer l ∈ [1, ..., L] during the inference of M on S (residual stream). We execute a separate inference of the model M with a different prompt T to decode the semantics embedded in these representations. This prompt T contains a special placeholder token "x" for the patching process: substituting its latent representation with a summary of the input's latent representations from the original inference. We specifically intervene during the model computation on prompt T by mapping the placeholder token's embedding with a weighted sum of the token embeddings of the input sentence obtained at the l-th hidden layer of the model inference on S. The execution of this patched inference generates then a structured text o l , containing a list of factual information. The procedure is repeated for the different values of l ∈ [1, . . . , L]. All the generated texts are then represented as a dynamic knowledge graph, with l being the graph's dynamic and temporal dimension.Essentially, the proposed framework converts the LLM internal vector space into a humanunderstandable semantic graph by collectively probing the encoded semantics of multiple token representations. The following further details the different procedure steps.Initially, we define a template for the model instruction, the source prompt S, encompassing three different semantic parts: (i) a system instruction describing how to accomplish the claim verification task, (ii) an input-output example to help the model generate the desired output, and (iii) the input sentence (Figure 3 ). The desired output is a structured text with two attributes: (i) a binary label indicating the truthfulness of the sentence, and (ii) a list of facts supporting such evaluation. The facts are represented as a conjunction of ground literals: asserted or negated predicates. For instance, the factual information necessary to evaluate the input claim "Edgar Allan Poe wrote Hamlet" can be represented as: AuthorOf(Hamlet, William Shakespeare) ∧ ¬AuthorOf(Hamlet, Edgar Allan Poe), where ∧ and ¬ indicate the logical conjunction and negation respectively. Using ground literals has two advantages: (i) formatting factual knowledge consistently for the subsequent knowledge graph generation, and (ii) stimulating the language model to associate and contrast factual information using its logic symbolic knowledge (De Smet et al., 2023) . Our preliminary experiments indicated that using a simple subject-predicate-object (SPO) triple representation leads to sub-optimal outcomes, resulting in more isolated and subject-focused 3 The terms latent/vector representations/space, embeddings, and hidden states are used interchangeably. information. The full prompt is shown in Appendix A. We then apply this template to a given input sentence and run the language model M on S, producing a structured output and storing the hidden states (H l ), the token representations across the model's layers.We denote the matrix H l as the latent representations obtained from the l-th hidden layer of the model's inference on the source prompt S. We then consider a separate inference of the model M on a different prompt, a target sequence of m tokens T = ⟨t 1 , ..., t m ⟩. This prompt mimics the source prompt to generate similar outputs but serves as a propositional probe. It decodes the semantics within the latent representations via activation patching. To perform the patching operation, we include a placeholder token, the character "x", within this prompt. We also augment the in-context examples (1 to 3) and reduce the system instruction to boost the model's in-context abilities during inference. The full prompt is shown in Appendix B. This patching operation consists of replacing the vector representation of the placeholder token ( h 1 k , where k is the position of x in T ) with a summary of the vector representations of the input sentence from original inference (M on S), leaving the other latent representations unchanged, and letting M proceed with the inference (Figure 4 ). Since activation patching is a vector-level technique, we formally define a matrix-to-vector mapping function f (H l , W) : R n×d → R d , parameterized by W, that computes a weighted sum of the latent representations of the input part I ⊂ S of the source prompt:f (H l , W) := L I ∑ i=1 w i h l i = hl (1)where the input part refers to the input sentence, the factual claim, and L I = |I| is its length. We set the weights W of the weighted sum by performing part-of-speech tagging4 on the input claim. Nouns and verbs are assigned a weight equal to zero to all but their end token which receives a weight of one (Figure 5 ), emphasising sentence's entities and predicates in this summary vector representation. We focus on end tokens based on Meng et al.'s (2022) study, which found that the model forms a subject representation at the final token of an entity name. Appendix G shows that including all tokens, especially stop words, is detrimental. Additionally, empirical experiments showed that using single tokens leads to meaningless texts for punctuation and single-word information for the last input token.Patching is then applied by replacing h 1 k with hl in the model's input embedding layer, as visually exhibited in Figure 4 . This model inference, patched with the summary input representation from l-th layer, generates a structured text o l , structurally equal to the one from the original model inference (M on S). By applying this procedure for all values of l ∈ [1, . . . , L], we produce structured outputs using all latent representations of M.Figure 5 : Example of the binary weights for the input's tokens I ⊂ S. These weights are then used to combine the tokens' vector representations via a weighted sum.We use a knowledge graph to represent the list of ground literals, the factual information, included in the structured output (Figure 3 ) generated by each patched inference (o l ∈ O | M on T ) and the original inference (M on S). We first turn literals into subjectpredicate-object (SPO) triples using simple rewriting rules:(¬)Relation(object 1 , object 2 ) → ⟨object 1 , (not)relation, object 2 ⟩ for binary predicates (¬)isProperty(object) → ⟨object, is(not), property⟩for unary predicates (2)Afterwards, we represent all the triples, yielded from o l , as a knowledge graph (Figure 6 ).We eventually concatenate all the graphs generated for the different values of l, creating a dynamic graph that exhibits the factual knowledge evolution across the model's layers.Figure 6 : Graph generation process from the ground literals to a knowledge graph. This process is performed for each o l ∈ O and the inference output. 
experiments: 
 This section showcases our framework on two experimental use cases: local and global interpretability analyses of the extracted factual knowledge. After describing our experimental setup in Section 4.1 and evaluating the effectiveness of the language model in Section 4.2, we present two interpretability analyses, a local analysis of the factual information of three distinct input claims in Section 4.3, and global analysis of the patterns in the factual knowledge resolution process in Section 4.4.To analyse the model behaviour in recalling factual knowledge to support or debunk input sentences, we prompt a language model with a collection of factual and common-sense claims sampled from two well-known claim verification datasets: FEVER (Thorne et al., 2018) and CLIMATE-FEVER (Diggelmann et al., 2020) . As in conventional mechanistic interpretability workflows (Conmy et al., 2024) , these inputs serve to elicit a model behaviour, unfolding the factual information an LLM holds internally (Section 1).Each claim is labelled as Supported (True), Refuted (False) or NotEnoughInfo. For instance, the claim "Charlemagne was crowned emperor on Christmas Day" from the FEVER dataset is classified as Supported, whereas "Berlin has a population of 4 million people" is classified as Refuted. These are all factual claims and a language model should rely on specific factual knowledge to evaluate their truthfulness. On the other hand, CLIMATE-FEVER, a dataset mimicking the FEVER methodology, compounds to real-world claims regarding climate change (Diggelmann et al., 2020) . Its claims may require more common-sense reasoning, and elicit subjective dichotomies. For example, the claim "Global warming is increasing the magnitude and frequency of droughts and floods" is labelled as Supported, whereas "Renewable energy investment kills jobs" is classified as Refuted.We initially filter the two datasets by (i) excluding the claims labelled with NotEnoughInfo, avoiding prompting the model with unverifiable sentences, and (ii) considering claims neither too short nor too long (35 ≤ |characters| ≤ 120). Afterwards, we randomly sample 1,000 claims from FEVER (Supported: 726, Refuted: 274) and 400 from CLIMATE-FEVER (Supported: 274, Refuted: 126). We use the LLaMA 2 language model (Touvron et al., 2023) in its 7-billion instruction-tuned version5 to showcase our framework. Appendix F compares the output generated by similar language models: the model's 13-billion version, the newest LLaMA 3 (AI@Meta, 2024) and the 7-billion Falcon model (Almazrouei et al., 2023) .Method. We here assess the effectiveness of the language model M to classify the input sentences. This helps to better understand whether the desired model behaviour is properly elicited: recalling helpful knowledge to evaluate claims' truthfulness effectively. The classification metrics consider the binary label within the structured text generated during the inference process (Figure 3 , M on S). Table 1 displays the metrics grouped for each target binary label: true (supported) and false (refuted). It also exhibits a self-consistency metric that measures the average consistency of the inferences' binary labels with those generated by the patched inferences (o l ∈ O | M on T ).Findings. The model achieves good performance in both datasets, reaching ROC AUC6 scores equal to 0.68 and 0.74 for FEVER and CLIMATE-FEVER respectively (Table 1 ). Table 1 reports also the F1 score. However, examining the model's performance reveals a significant imbalance, especially on the FEVER dataset. It has high precision and low recall for the claims classified as true, and low precision and high recall for those classified as false (Table 1 ). Intuitively, this may be because falsifying a claim often requires less factual knowledge than is needed to prove it true. On the other hand, the CLIMATE-FEVER dataset demonstrates balanced recall performance for both classes, with a precision reduction in the false ones. This suggests the model encounters comparable difficulty in verifying or debunking claims when this mainly depends on common-sense reasoning (Section 4.1) while erroneously classifying more claims as false. The confusion matrices are reported in Appendix D. A dichotomy between the two classes is also observed in the self-consistency metric: when the inference generates a true label, on average, almost half of its hidden layers generate the same binary label, whereas just about ten percent of the layers' labels coincide with the inference prediction for the false label (Table 1 ). Method. We investigate the factual information encoded within the model's latent representations of three distinct input claims. Claim A ("Renewable energy investment kills jobs") is from the CLIMATE-FEVER dataset, and requires common-sense reasoning. In contrast, Published as a conference paper at COLM 2024 claim B ("Charlemagne was crowned emperor on Christmas Day") and claim C ("George Lucas founded a visual effects company") are from FEVER and require multi-hop reasoning. Figure 7 exhibits five snapshots of the dynamic knowledge graphs related to these three inputs, while Appendix C shows the comprehensive graph for claim B. Findings. All claims in Figure 7 show word-based factual information in the early layer L 3 . However, while claims B and C accurately present information about their entities, claim A incorrectly exhibits information about the last in-context example (The Beatles, Appendix B). In Section 4.4, we hypothesise that example leakage may occur when the probed latent representation lacks factual knowledge, shifting the model's focus to the provided context. Claim A then unfolds an interesting knowledge evolution: Layer 7 encodes factual knowledge concerning Energy, whereas Layer 17 and Layer 19 create a clear dichotomy between entities that are Job Killer (e.g., Coal and Nuclear Energy) and Job Creator (i.e., Energy Efficiency Upgrades). Although this can be common-sense reasoning, it might unveil biases in this energy-related dichotomy. On the other hand, Claim B unveils a swap in the entity representation in the middle layers. Layer 9 correctly encodes factual knowledge concerning the subject entity (Charlemagne). Layer 14 then confuses and swaps it with Charles V (another former Holy Roman emperor), yet succeeds in multi-hop reasoning by connecting Christmas Day, its actual date, and the emperor's coronation. The third claim exhibits another error related to internal knowledge representation. Layer 9 correctly encodes George Lucas as Founder Of the company Industrial Light & Magic, while treating its acronym (ILM) as a different entity. Subsequently, layer 19 associates the information of being a special effects company only with the acronym, while separately connecting its full name to its founder, George Lucas. This disjoint association and entity duplication lead to a multi-hop reasoning error in the inference (Figure 7 ). While unifying the company's representations, the model fails to reconnect George Lucas as its founder. This knowledge-recalling issue might stem from attention mechanisms or catastrophic forgetting.Method. This analysis reveals patterns in the evolution of factual knowledge. We seek behaviour changes across the hidden layers, thus identifying transaction points in this latent evolution. We examine the knowledge graphs generated by the latent representation from each hidden layer by calculating graph-level similarities and identifying cross-layer similarities. We initially compute node embeddings for each dynamic knowledge graph derived from a subset of over five hundred FEVER dataset inferences. We consider input claims with character lengths falling within the first and third quartiles. We use the Multi-Scale Attributed Node Embedding method (MUSAE, Rozemberczki et al., 2020; 2021) to generate node embeddings Z ∈ R n×d for each graph, where n represents the number of graph nodes and d is the embedding dimension (d = 4096). Next, we assess the semantic correspondence between two graphs (G and G ′ ) using a custom graph similarity measure on the node embeddings: (i) computing pair-wise node similarity, (ii) identifying the highest similarity matching for each node of G, and (iii) averaging these similarities:EQUATIONwhere n G and n G ′ are the number of nodes of G and G ′ respectively, and sim cos is the cosine similarity. To investigate patterns in the evolution of the encoded factual knowledge, we focus on the similarities of the graphs generated using the latent representations of two consecutive layers: o l and o l-1 . Appendix I provides layer-wise cosine similarities. We repeat this process for all the dynamic knowledge graphs and report aggregated results in Figure 8 . Afterwards, we employ the MeanShift clustering algorithm7 (Cheng, 1995; Pedregosa et al., 2011) on the similarity data. The bandwidth hyper-parameter, which affects the cluster granularity, is estimated using the 25th percentile of the sample data. This clustering procedure identifies layers exhibiting similar changes in the evolution of their factual knowledge (colours in Figure 8 ), indirectly spotting transaction points.Graphs from the l-th layer Findings. We validate the transaction points unfolded in Figure 8 by examining the type of factual information decoded from the cluster items across several individual inferences (Appendix E). We observed that early layers (L 1 : L 3 , cluster B) focus on the syntax and entity resolution process, as already demonstrated by Ghandeharioun et al. (2024) . We also discovered a slightly different evolution pattern when tokens represent partial or complete words. When tokens correspond to complete words, such as "Germany" and "Robin" (Appendix E), the model encodes factual information related to the entity or its semantic context, for instance, IsCountry(Mexico) and SuperHeroOf(Robin, Batman). However, in the case of incomplete words, the model accomplishes entity resolution using encoded word-based representations. For example, merging the token representations of "jack" and "eman", the model tries to decode the subject entity as "Jackman", yet the original subject was "Bojack Horseman" (Appendix E). Further, these early layers produce valid outputs (structured texts) in only 40% of the considered inferences. In contrast, the latent representations from the 4 th hidden layer (cluster C) generates valid factual information regarding the subject's semantic context in 75% of the inferences. For instance, WasQueen(Mary Queen of Scots) for the original subject "Empress Matilda" or IsHistoricalFigure(Charles) for "Charlemagne" (Appendix E). Thus, we empirically show that the model exhibits minimal to no factual knowledge in these early layers.Middle layers (L 5 : L 21 , cluster D) exhibit comprehensive factual knowledge concerning the subject entity, with an evolution towards the requested factual information, for example regarding whether Empress Matilda moved to Germany (claim 1 in Appendix E): L 7 = {. . ., BornIn(Empress Matilda, England)}, L 15 = {. . ., DiedIn(Empress Matilda, England)}, and L 18 = {. . ., MovedTo(Empress Matilda, Germany)}. These middle layers also exhibit an interesting pattern: whenever the model achieves comprehensive factual knowledge for a given entity, it moves its attention towards another entity in the sentence (or a semantic entity neighbour), and represents other contextualised factual information. For claim 2 in Appendix E, for example, the model represents the factual information concerning "Robin" in layer 4, then concerning "The Joker" between layers 6 and 17, and eventually further factual information concerning "Robin" between layers 18 and 19.On the other hand, we noticed a decline in the factual knowledge decoded from the latent representations of the bottom layers (L 27 : L 32 , cluster A) as well as the initial layer. Over 90% of the inferences patched with these layers yield unstructured texts, often containing nonsensical text or references to the last in-context example (The Beatles, Appendix B). The latter leads us to speculate that the model's attention may have shifted towards the in-context examples in these last layers. Indeed, this seems supported by the analysis of the attention matrices across layers (Appendix H), which shows a slight concentration of attention towards groups of tokens within the in-context example tokens in the final layers. As for the reason behind such an attention shift, we elaborate on two speculative hypotheses: (i) it may originate from the limited amount of factual knowledge encoded in these latent representations, and (ii) the previously mentioned store-and-seek pattern might influence this lack of encoded knowledge. As a result, we speculate that the beginning of this attention shift corresponds to the conclusion of the factual knowledge resolution process, which could be influenced by factors such as the number of contextually relevant entities or the extensiveness of their semantic domain. For example, the beginning of output degradation is differentiated for the three inputs which pertain to increasingly specific domains: an empress's life (claim 1 in Appendix E), a TV show's creator (claim 3), and a comic book event (claim 2). It respectively begins at the 24 th , 22 nd and 20 th layers. Interestingly, for the broader domain (claim 1), the model represents factual information about a different empress between the 20 th and 23 rd layers, validating this transition point. 
conclusions: 
 This work proposes an LLM-based framework to study the process of factual knowledge resolution from LLM latent representations. This framework decodes the semantics embedded within the LLM vector space, in the form of factual information, via activation patching exclusively, avoiding reliance on training or external models. This enables richer analyses not easily accessible with other probing techniques and enhances the understanding of LLMs' factual knowledge and layer-wise processing. Our two experimental use cases show how the proposed framework provides novel insights into the LLMs' factual knowledge resolution process both locally and globally. The graph structure enhances local interpretability by highlighting entity centrality in LLM reasoning, from the subject's factual information and multi-hop reasoning to representation errors causing erroneous evaluation of the input claim. Conversely, the global analysis reveals patterns in this underlying process, combining established LLM phenomena, such as early layers focusing on syntax, with new findings, like word-based knowledge evolving into claim-related facts. Although the framework applies to other instructed-tuned language models, these insights may vary depending on the model's architecture and task. Future work can extend this framework, for example, by varying the considered tokens across the model's inference to study information flow or conduct further graph-related analyses on the outputs. Finally, the absence of ground truth knowledge limits the evaluation of the generated factual information to a qualitative analysis of its relevance to the input's evaluation. Future research can focus on quantifying this relevance for input and claim verification. 
 
 Appendix: You are a journalist with expertise in fact-checking. Your role is to evaluate the truthfulness of factual claims. To uphold journalistic integrity, you must produce a report containing a binary assessment and all the factual information that supports your evaluation. Each factual information should be presented as zeroth-order logic propositions.George W. Bush won a presidential election 
 {"label": true, "facts":["isPolitician(George W. Bush) ∧ isFormerUSPresident(George W. Bush)","ParticipatedIn( 2000United States presidential election, George W. Bush)","BecamePresidentOf(United States of America, George W. Bush)"]} 


 $INPUT 
Figure 9 : Template for the source prompt S. This is applied for each input claim (INPUT).You are an assistant with expertise in fact-checking. Your role is to assess claims using zeroth-order logic propositions.Berlin is the capital of Germany Cluster 1: "Empress Matilda moved to Germany as a child" 2: "Robin was murdered by the Joker in a 1989 book" 3: "Bojack Horseman's creator is also American"A "Here is the updated output:" "I apologize, but I'm a large language model, I cannot provide you with a book called "robin". As a responsible AI language model...""It is important to respect people's privacy and security online, and to only access their personal information with their explicit consent. Attempting to hack... " I apologize, but "Jaman" is not a valid name. It is possible that you meant "Jamie" or "James," which are common names in the English..."L2 "I apologize, but "Mildilda" is not a valid name. The correct spelling is "Mildred"."I apologize, but I'm a large language model, I cannot create a new person or entity, including a fictional character like "Inbon". However..." "I apologize, but "Jackman" is not a valid name. Could you please provide a correct name?" L3 "Here is the updated response:" IsPerson(Robin) ∧ SuperheroOf(Robin, Batman) "I apologize, but "Johndeman" is not a valid name. "To provide a more accurate answer, you could say: "The Beatles were a British rock band, with members" L26 "To provide more accurate information, here are some additional facts: "To provide more accurate information, here are some additional facts:" "To provide more accurate information, here are some additional facts: L27 "To provide more accurate information, here are some additional facts:" "To provide more accurate information, here are some additional facts:" "To provide more accurate information, here are some additional facts:" Table 4 : Comparison of the generated text from the inferences patched with a summary representation generated using different token sets. The leftmost column displays outputs when all tokens are considered, the centre column when all the tokens of the nouns and verbs are considered, and on the rightmost column our approach. EQUATIONWhere nodes (entities) and edges (relations) change over time.https://spacy.io/models/enhttps://huggingface.co/meta-llama/Llama-2-7b-chat-hfReceiver Operating Characteristic Area Under the Curvehttps://scikit-learn.org/stable/modules/generated/sklearn.cluster.MeanShift 


Decision: The submission builds upon very recent and impactful work in LLM interpretability and presents a novel and useful extension, in particular in that unlike the recent Patchscopes it includes full relations. The negative reviews seem to have several important misunderstandings which are resolvable by a more careful look at the submission and the authors' response. The submission has sufficient amount of empirical results complemented with a qualitative analysis in its current form, and the promised ablation results and clarification improvements are doable and will further improve the quality of the work. I recommend acceptance.

Reviews: review: paper_summary: paper_summary: **Paper Summary**

This work describes a new method for attempting to provide some interpretability in the context of LLMs when verifying factual claims. More specifically, the work describes a way to construct a temporal knowledge graph, derived from LLM hidden representations, to better visualize working state and reasoning of the LLM during inference. The mechanism uses a few ideas including asking the model to generate a chain of predicates as rationale for claim verification, along with using activation patching to translate hidden representations of different layers of the LLM into knowledge triplets. The work describes mostly qualitative analysis of constructed temporal knowledge graphs when evaluating LLaMA-2 model on the FEVER and CLIMATE-FEVER datasets.

**Review Overview**

Overall, it is not possible to recommend this work for publication in its current form for two main reasons: (1) I have several concerns about the soundness of the techniques employed (that is, the interpretability techniques can be trusted to correctly reflect internal state of the model) and (2) the analysis done using the proposed techniques on FEVER and CLIMATE-FEVER are a bit unclear or redundant given prior work.

In terms of originality, given that this is a heavily explored area, this proposal in this paper adds only a small delta to recent interpretability techniques of LLMs. The “temporal” (layer-wise) aspect of the work is not novel (e.g., https://aclanthology.org/P19-1452/ from 2019 although not explicitly focused on factuality). Augmenting LLM rationales with first-order-logic is also interesting but not quite new (e.g., https://aclanthology.org/2023.findings-emnlp.416/ from EMNLP 2023 and https://openreview.net/forum?id=qFVVBzXxR2V from ICLR 2023).

In terms of clarity of presentation, I found the paper generally clear and well written.

A few potential useful references that are missing:
Dissecting Recall of Factual Associations in Auto-Regressive Language Models - https://aclanthology.org/2023.emnlp-main.751/
https://www.alignmentforum.org/posts/iGuwZTHWb6DFY3sKB/fact-finding-attempting-to-reverse-engineer-factual-recall
strength_weakness: strength_weakness: As mentioned in the summary, the results of the work are still a bit immature. The paper is not thorough in convincing that the techniques used are faithfully representing the internal workings of the LLM and are not, for example, post-hoc explanations.

Specifically:

- One important component of the proposed system is the first-order-logic derivation that the model produces when also producing a verdict of whether a statement is factual. There was never any analysis of whether the logic produced was faithful to the end result. What evidence do we have that the first-order-logic produced is not a hallucination or post-hoc explanation by the model? How faithful is it to its response?

- It is also unclear how faithful the patching mechanism works. It would seem that averaging the output of many tokens (all first-order-logic tokens?) into a single representation would constitute a potential loss of information (as described in Section 3.2). If the first-order-logic contains several entities and relations can we assume that the averaged representation of the tokens can recover the information? Presumably this can be tested in some way (e.g. just for the last/output layer where you know the ground truth first-order-logic and the graph that could be constructed from it),

- The analysis in Section 4 is a bit limited. Aside from general trends in layer-depth representation  (which has been studied previously), there are no new insights derived from the results. It makes me wonder if this may be a reflection of how useful temporal KGs are in practice. How would they be useful to debug specific factuality problems in practice?
questions: questions: Besides the comments above, I few more minor questions:

- There are two prompts used in the work (source S, and target T) that are almost the same. I would have expected them to be identical since the patching exercise is attempting to reconstruct the internals of the original inference. Why are these prompts not the same?

- In Section 3.2, the way to construct a summary representation uses some heuristics (averaging, weighting certain terms more or less). Did you try different representations to transfer from inference to patching? What was the impact of those choices? Why not take the entire sequence instead of trying to summarize into a single representation?

- In the results, there is a mention that “ILM” and “Industrial Light and Magic” are different nodes in the knowledge graph. This seems to be just an aliasing problem (that is, the model is likely internally representing the two surface forms almost the same). Why did authors think that this aliasing is causing a multi-hop reasoning problem? How do we know the model internally is not representing these concepts sufficiently similarly?
paper_summary: paper_summary: Authors propose an approach to understand the inner workings of Large Language Models. To do so, they chose the task of factual knowledge verification. The idea behind factual knowledge verification is to utilize claims from verification datasets like FEVER and CLIMATE FEVER, converting them into a restricted first-order logic form. They create a prompt and pass it through the model to predict the logical form as well as the label.
strength_weakness: strength_weakness: * Evaluation: While the work is novel, the performance of the proposed approach is nearly similar to or slightly better than a coin toss (50%) since the model's ROC AUC performance is close to 0.5 or 0.6. Given this, it's difficult to determine if the internal workings of LLMs are appropriately evaluated with proposed approach.

* Missing design choices: Throughout the paper, there are several instances where authors have made design choices without explanation. For example, why the study is limited to LLama and not other kinds of models. Is the observation visible only in Decoder-type models or can it also be seen in Encoder-Decoder type models like FLAN? Additionally, what was the intuition behind removing "Not Enough Info"? Moreover, what is the reason to consider only FEVER or CLIMATE FEVER ? 

* The approach uses restricted First Order Logic to represent the claim. The language used to represent the claim is restricted as the relations are binary and not n-ary. Hence, they do not accurately represent each of the facts mentioned in the section. The datasets used in FEVER and CLIMATE FEVER are full of sentences containing n-ary relationships among the entities. Therefore, a better representation would benefit the work.
questions: questions: Suggestion

* Remove the words "Temporal" and "knowledge graph." The term "temporal knowledge graph" is misleading as it triggers the reader to think in a different angle, making it hard to relate to the present approach. Also, the term "temporal" is used loosely, as the layers of LLM; it's better to consider three tiers of layers in LLM: Bottom, Middle, and Top.
paper_summary: paper_summary: This paper presents a method for probing latent representations of (local) knowledge graphs in an LLM, in a way that is amenable to both local understanding – the representation of a particular statement – and global analysis – looking at how groups of layers behave across examples. The method builds on previous activation patching work from Ghandeharioun et al. 2024 and Zhang and Nanda 2023, but uses more complex prompts to decode full relational triples from the representations of a factual (or false) statement.

The authors validate both their base model inference (M on S) and the patching (M on T) on the FEVER and CLIMATE-FEVER datasets, measuring both standard accuracy as well as self-consistency as a way of validating the decoded facts. They present a variety of interesting qualitative findings, both at the local level - tracing the evolution of a model’s knowledge graph around a single claim - and globally by clustering graphs across layers and identifying common patterns, such as a transition from entity resolution in earlier layers to representing factual knowledge about these entities in the middle layers.
strength_weakness: strength_weakness: I have some reservations about the complexity of the measurement technique, as the complex prompts (particularly on the target side) and relatively long-form output leave room for potential bias or skew in the results that may be due to the measurement technique rather than inherent properties of the model. This has been discussed in the literature w.r.t. traditional probing models (e.g. https://arxiv.org/abs/1909.03368), and I worry about the same issues appearing here - for example, the authors discuss at least one case of this where there appears to be leakage from a few-shot example in the target prompt. I would encourage the authors to discuss this more, as this can often be addressed by experimental design - such as including appropriate controls or baselines, marginalizing over different prompts, or corroborating specific findings using simpler methods.

The global analysis also may suffer from this, as the graph node embedding and clustering makes the findings a few more steps removed from the underlying source-side model representations. I wonder if some observations such as the shift from “Robin” to “The Joker” and back to “Robin” (2nd paragraph on page 8) could be artifacts of the discretization steps involved here.

That being said, I think this is at least somewhat orthogonal to the main contributions, and would like to still see this paper published.
questions: questions: - It's not super clear until later in the paper (3.3) what the “temporal” dimension was that the title refers to. Maybe mention in the abstract and introduction that this means over layers of the model?

- I’d be curious to see ablations on the pooling across tokens here. On one hand, it seems like the noun/verb last-token approach gets most of the content words in the input sentence, and so I wonder if it would be much different from just pooling across all tokens. On the other hand, since the model processes left to right, I wonder if it would work to just take the representations of the last token in the claim?

- Did you experiment with any alternative decoding strategies, such as the model to decode the SPO triples directly rather than the relation form? I wonder if these would be easier to generate, as they seem closer to natural language.

- “In the form of ground predicates and without relying on external models or training processes” - I would hesitate a bit to claim there is no training involved, since the method relies on few-shot examples in the prompts. But I don’t think this detracts from the main contributions here.
paper_summary: paper_summary: This paper proposes a framework to understand how Large Language Models (LLMs) recall and use extensive factual knowledge. This paper uses the activation patching method to perform interpretability analysis on two datasets, FEVER and CLIMATE-FEVER. The local interpretability analysis exposes different latent errors from representation to multi-hop reasoning errors. The global analysis uncovered patterns in the underlying evolution of the model's factual knowledge.
strength_weakness: strength_weakness: 1. The method in this paper is somewhat limited, and how to expand it to a broader range of factual knowledge research is not yet clear.
questions: questions: 1. Can the method described in this paper be extended to other tasks involving factual knowledge in Large Language Models (LLMs), especially in AI-assistant dialogue scenarios?

Comments: title: Authors' additions
Authors' Response to Reviewers' Feedback and Proposed Improvements
comment: Thank you for your response - I appreciate the clarifications, particularly around token pooling and decoding strategies.

> Simplifying the target prompt might not significantly affect the findings, such as reducing examples or removing the binary label from the output. On the other hand, training an external probe might be non-trivial, given our desired output: a list of factual information. 

I don’t think this necessarily means training an external mlp or linear probe, as you do have a strong argument that those can’t decode the semantic structure you are interested in. But this doesn’t mean you couldn't do some ablations of the measurement technique, for example to study variability / sensitivity to details of the probe design. 

> Conversely, training involves feeding data into a model and adjusting its internal weights through a cost function and backpropagation.

I think your reviewers are aware of this definition :) I did not intend to split hairs on the definition of “training” vs “learning”; but rather that it would not be appropriate to imply that the method does not need at least some amount of labeled data - or particularly, that it’s not sensitive to the way this data (in the prompt) is constructed.
*Layerwise aspect is not novel*: ... For example, previous studies have analysed simpler objects than factual knowledge such as *factual associations* (Meng et al., 2022 and Geva et al., 2023), *semantic concepts* (Nanda et al., 2023 and Anthropic, 2024) and *entity resolution* (Ghandeharioun et al., 2024).

*Patching and prompts*: During the inference on prompt T, the patching technique replaces the vector representation of the placeholder token with the input claim's summary vector representation from the original inference (Figure 4). This patched inference interprets the semantics within this injected vector representation. Essentially, it translates the unknown vector space used internally by the LLM (and altered by merging multiple token representations) into human-understandable semantics. This process is more similar to translation rather than reconstructing the original inference.
We appreciate the thoughtful and positive comments from all reviewers. You have recognised that our framework is *an elegant extension* of the recent work by Ghandeharioun et al. (ICML 2024, XVV9), *enables richer analysis not easily accessible with other probing techniques* (XVV9); *advances the scope of better understanding internal workings* (GvEw), *representations* (XVV9), *factual knowledge* (3hbG), and *layer-wise processing* (XVV9) in LLMs; provides *scalability for research on interpretability* (yHaY); and the paper is *well-written* and presents the problem setting, the proposed technique and results clearly (3hbG).

We believe that addressing the reviewers’ feedback and questions has significantly enhanced the quality of our manuscript. In the final version, exploiting the additional page, we will improve the following aspects:
### Clarification of the work's objective, the roles of the logic representation and the NLP task
We will provide the following clearer explanation of the objective of our work as well as the purpose of the NLP task and logical representation within our framework (GvEw, 3hbG) in the abstract, introduction, and methodology sections:
- Using latent representations of LLMs, our work uses the NLP task of factual knowledge verification to elicit a model behaviour for probing: exposing the factual information that an LLM holds internally when evaluating the truthfulness of an input; with no ground truth available. We thus neither predict the logical form of claims (GvEw) nor the model outputs (3hbG). 
 - On the other hand, we mainly use the logical representation to consistently format the factual information, which supports the evaluation of the input claim, for the subsequent graph generation. Ground predicates, in first-order logic, format the facts related to claims rather than converting the input claims (GvEw). For example, the factual information and the ground predicate “AuthorOf(Hamlet, William Shakespeare)” for the input claim “Edgar Allan Poe wrote Hamlet” (Section 3.1).
### Graph utility
Sections 1 and 4 will better present the advantages of our graph representation, from highlighting entity and fact centrality in LLM reasoning via local analysis to revealing trends in factual knowledge resolution in global one (3hbG, XVV9).
Specifically, we will better introduce the result section by further stressing its showcase purpose, presenting use cases of actual outputs of our framework, rather than as our main contribution. Further works can leverage our framework to conduct other graph-related analyses on LLM’s latent representations.
### Clarification concerning the chosen datasets 
Section 4.1 will clarify that the FEVER and CLIMATE-FEVER datasets are used to collect factual and common-sense claims for eliciting a behaviour to analyse (GvEw), as in conventional mechanistic interpretability workflow (Conmy et al., 2023). 
These input claims, already verified with publicly available information (GvEw), are exploited to trigger a language model to recall factual knowledge that supports or debunks them.
### Clarification concerning the classification performance
Section 4.2 will better explain the purpose of analysing the classification performance of model inferences. We will rename the section and remove the latent representations’ performance (GvEw) to reduce misunderstandings for readers.
### Confusion with the term “temporal”
- In Sections 1 and 3, we will better introduce and rationalise the temporal dimension of our graph representations. The sequential nature of hidden layers in a transformer-based model allows segmentation within a discrete temporal dimension during inference: layer *t* at time *t*, layer *t+1* at time *t+1*, and so on (GvEw). 
- However, based on reviewers' feedback about the confusion and non-conventional use of this term (GvEw, XVV9), we will adopt the more general term “dynamic” in the final version of our manuscript as Harary and Gupt (1996).
### Explanation and ablation concerning the token strategy for the summary representation
We acknowledge the reviewers' questions about the rationality, effects, and alternatives of our strategy for the summary representation (3hbG, XVV9). Section 3.2 of the final manuscript will detail the experimental results that led us to propose such a token pooling strategy: 
- We will explain the need for pooling (3hbG) and the unsatisfactory results of using single tokens, such as meaningless texts from punctuation or model’s special tokens, and single-word information for the input’s last token. 
- An appendix will include an ablation study with different multi-token strategies, showing outputs when considering all claim tokens or complete words.
### Integration with a preliminary comparison with other models
We will include a visual comparison of outputs generated by other instruction-tuned LLMs in a tabular appendix, testing the latest LLaMA3 (8-billion) as well as LLaMA2 (13-billion) and Falcon (7-billion).