Understanding and Patching Compositional Reasoning in LLMs (2024)

Abstract: LLMs have marked a revolutonary shift, yet they falter when faced with compositional reasoning tasks. Our research embarks on a quest to uncover the root causes of compositional reasoning failures of LLMs, uncovering that most of them stem from the improperly generated or leveraged implicit reasoning results. Inspired by our empirical findings, we resort to Logit Lens and an intervention experiment to dissect the inner hidden states of LLMs. This deep dive reveals that implicit reasoning results indeed surface within middle layers and play a causative role in shaping the final explicit reasoning results. Our exploration further locates multi-head self-attention (MHSA) modules within these layers, which emerge as the linchpins in accurate generation and leveraing of implicit reasoning results. Grounded on the above findings, we develop CREME, a lightweight method to patch errors in compositional reasoning via editing the located MHSA modules. Our empirical evidence stands testament to CREME’s effectiveness, paving the way for autonomously and continuously enhancing compositional reasoning capabilities in language models.

heading: 1 Introduction
text: Compositional reasoning stands as a pivotal mechanism, unlocking the ability of learning systems to decompose complex tasks into manageable subtasks and tackle them step-by-step (Lu et al., 2023; Lake and Baroni, 2023). Despite the revolutionary impact of Large Language Models (LLMs) on the NLP landscape, they struggle at basic compositional reasoning tasks (Dziri et al., 2023). This shortcoming is specifically highlighted by Press et al. (2023), who brought attention to the concerning “compositionality gap” in the realm of question-answering tasks. It was observed that there is a substantial failure rate of ∼ 40% in
* Corresponding authors.
two-hop compositional queries, even when they can successfully answer the individual single-hop queries that make up the two-hop question. Recent attempts improve the compositional reasoning capabilities of LLMs by making them decompose the question and explicitly output the step-by-step rationale with carefully crafted prompting strategies developed by experts (Wei et al., 2022; Zhou et al., 2023). However, understanding the inherent mechanism of compositional multi-hop reasoning inside LLMs and enabling them to autonomously rectify their compositional reasoning errors and continuously improve over time remain largely underexplored.
This work, therefore, sets out to firstly delve into the specific failures to understand (RQ1) what accounts for these compositional reasoning failures and (RQ2) which parts of the LLMs are responsible for them, and secondly develop strategies for patching these failures. Our initial step involves
9668
an analysis of a very recent dataset comprising compositional two-hop knowledge queries (Zhong et al., 2023), selectively examining the cases where LLMs fail despite successfully answering the constituent single-hop queries. To ensure our findings and methodologies offer broad applicability, our analyses utilize two widely-used open-sourced LLMs: OpenAlpaca-3B (Su et al., 2023b) and LLaMA-2-7B (Touvron et al., 2023). Through meticulous examination of the failure instances, we identify three prevalent types of errors. Utilizing the Logit Lens tool (nostalgebraist, 2020), each error type highlights a critical shortfall in generating or leveraging the implicit reasoning result necessary for the explicit reasoning result1. This gap is particularly concerning as it contrasts sharply with the intuitive two-hop reasoning process inherent to human cognition. An illustrative example of compositional reasoning error is depicted in Figure 1, where the model incorrectly concludes its reasoning without properly generating and incorporating the implicit reasoning result.
The above observations motivate our further empirical inquiry to answer the first question of what accounts for these failures, from the perspective of whether LLMs are indeed aware of implicit reasoning results during compositional reasoning. We inspect inner hidden states of LLMs via Logit Lens, from which we observe that implicit reasoning results not only manifest within the LLMs’ intermediate layers but also tend to precede the generation of explicit reasoning results, often emerging statistically earlier. Building on this, we further explore the relationship between implicit and explicit reasoning results through an Intervention (Pearl, 2001; Li et al., 2023a) experiment, providing compelling evidence that the emergence of implicit reasoning results within LLMs plays a causative role in the generation of explicit reasoning results.
The next question is, regarding RQ2, in which modules LLMs generate implicit reasoning results? Leveraging causal mediation analysis (Meng et al., 2022; Stolfo et al., 2023), we present both a compositional query and its corresponding second-hop query to the LLM, resulting in the generation of two distinct computation graphs. We then intervene the computation graph G1, associated with the compositional query, by replacing the output of a single module with its counterpart from the
1Compositional two-hop queries require two-hop reasoning: implicit reasoning result is the first-hop reasoning result; explicit reasoning result is the second-hop reasoning result.
second-hop computation graph G2. By identifying the modules whose replacement results in a significant enhancement in the predictive probability of the explicit reasoning result, we are able to locate several specific outputs from the Multi-Head Self-Attention (MHSA). Intriguingly, the layers pinpointed through this approach show a strong correlation with those identified in preceding Intervention experiments. This congruence reinforces the hypothesis that implicit reasoning results are not only present but are actively consolidated and utilized within these specific layers of the LLM.
Grounded on our findings into RQ1 and RQ2, we develop CREME (Correcting Compositional REasoning via Model Editing), a light-weight model-editing method to patch errors in compositional reasoning. CREME follows Santurkar et al. (2021); Meng et al. (2022) by regarding the output matrix of the located MHSA, W lo, as a linear associative memory. To implement CREME, we designate the input to W lo in the computation graph G1 as k∗ and the output from W lo in G2 as v∗. We then proceed to insert the pair (k∗, v∗) into W lo, ensuring that this insertion disrupts existing memories within W lo as minimally as possible. This objective is achieved by solving a convex optimization problem, which strikes a nuanced balance between the integration of new corrective information and the preservation of existing knowledge.
Our main contributions and takeaways are summarized below: (1) successful compositional reasoning within LLMs hinges on its awareness of generating and leveraging implicit reasoning results; (2) MHSA modules in the middle layers (18/19-th layer) are significantly in charge of properly generating and leveraging implicit reasoning results; (3) by leveraging the second-hop computation graph as a reference for editing the located MHSA modules, CREME proves to be highly performing, on correctly answering not only the query used for editing W lo but also the paraphrased queries and other compositional queries sharing the first-hop knowledge as well as maintaining little effect on irrelevant queries2.

heading: 2 Background & Notation

heading: 2.1 Logit Lens
text: Logit Lens (nostalgebraist, 2020) is a widely used for inspecting hidden states of LLMs (Dar et al.,
2Implementation will be available at https://github. com/Zhaoyi-Li21/creme.
2023; Geva et al., 2023; Katz and Belinkov, 2023; Sakarvadia et al., 2023). The key idea of Logit Lens is thus to interpret hidden states in middle layers of LLMs via projecting them into the output vocabulary space with the LM head Wu. When presented with a specific hidden state htl and a set of target tokens Ttgt, the Logit Lens is given as follows:
L(htl , Ttgt) = 1 |Ttgt| ∑
k∈Ttgt ptl [k], (1)
ptl = softmax(v t l ) = softmax(h t lWu), (2)
where L(htl , Ttgt) measures how much information around Ttgt is contained in htl . Note that typically there are multiple tokens in Ttgt and we separately calculate the probabilities for all of these tokens and adopt their mean value to represent the whole Ttgt, similar with (Chanin et al., 2024). There are other works (Sakarvadia et al., 2023; Yang et al., 2024) use the first token in Ttgt to represent it, which sometimes can bring loss of information in Ttgt and additional conflicts when processing targets like “United Kingdom” and “United States” as well.

heading: 2.2 Compositional Reasoning and Dataset
text: Compositional knowledge refers to knowledge items that are the compositions of several singlehop sub-knowledge items. Compositional reasoning refers to the ability to answer the queries on compositional knowledge (e.g., verbalized in format of QA or Cloze-Test) via a step-by-step reasoning process. We denote a single-hop knowledge as a triple (s, r, o), where s, r, o represents subject, relationship and object respectively. The composed compositional two-hop knowledge is denoted as (s1, r1, o1)⊕ (s2, r2, o2) where subscripts 1 and 2 represent the first-hop and second-hop sub-knowledge (requiring o1 = s2 so that they can compose together). The dataset D (Appendix B) we used in this paper is sourced from Zhong et al. (2023). For each datum in D, it contains: (1) the
compositional query on the compositional knowledge (s1, r1, o1) ⊕ (s2, r2, o2), (2) the first-hop query on (s1, r1, o1), (3) the second-hop query on (s2, r2, o2), and (4) the implicit reasoning result o1 and the explicit reasoning result o2. By way of example, the first-hop query is “What is the sport associated with (r1) Giorgio Chinaglia (s1)? association football (o1)”, the second-hop query is “What is the home country of (r2) association football (s2)? England (o2)” and the compositional query can be verbalized as “What is the home country of (r2) the sport associated with (r1) Giorgio Chinaglia (s1)? England (o2)”.

heading: 3 Analyzing Compositional Reasoning Errors
text: Grounded on the observation of Press et al. (2023), we dive into the compositional reasoning failures: we identify three types of common errors among such failures and attribute the cause of these common errors to the failure of generating implicit reasoning result properly via inspecting hidden states.
Three types of Common Errors We query LLMs with all of compositional queries and the corresponding single-hop queries in D. We filter out two subsets of D: Dsingle and Dgap. For each datum (s1, r1, o1)⊕ (s2, r2, o2) in D, Dsingle contains the datum where the both of (s1, r1, o1) and (s2, r2, o2) are successfully answered. Among Dsingle, Dgap contains the datum where the answer for the compositional queries (s1, r1, o1) ⊕ (s2, r2, o2) are mis-predicted.3 In our analysis of Dgap, we have discerned a few common patterns shared among a substantial portion of the failures. Consequently, we have delineated three predominant types of errors, each characterized by distinct features, as outlined below. Distortion: LLMs fail to effectively generate implicit reasoning results in the reasoning process. The predicted answer for the first example in Table 1 is either United Kingdom or Italy. Considering both as countries (corresponding
3Please find details in Appendix D.2.
to nationality (r2)), we conclude that the information about Donna Summer (o1) distorts in middle hidden states. Incomplete Reasoning: LLMs directly output the first-hop reasoning result (o1). In the second example of Table 1, LLaMA-2 outputs France (o1) while the correct answer requires further reasoning. the head of state of (r2) France (o1) is Emmanuel Macron (o2). Hasty Answer: LLMs predict the result without carefully reasoning. We further subdivide this type of errors into two categories: I: LLMs finally predict a close result based on the implicit reasoning result. For the third example in Table 1: LLMs predict Los Angeles or New York, both of which are famous city in the U.S.A., implying that LLMs manage to generate the implicit result (o1:U.S.A.) while fails to incorporate “the capital of” (r2) to generate final result o2. II: LLMs take short-cut instead of step-by-step reasoning, leading to incorrect answers. Consider the fourth example in Table 1: the correct reasoning process should be (1): the sport associated with (r1) Giorgio Chinaglia (s1) is association football (o1); followed by (2): the home country of (r2) association football (o1) is England (o2). However, LLMs erroneously attribute Italy as the answer. This misstep is attributed to LLMs’ tendency to directly associate Giorgio Chinaglia (s1) – noted for his Italian nationality – with the home country of the sport (r2). We calculate the proportions for each type of errors observed in the 200 two-hop examples of MQuAKE dataset (Zhong et al., 2023) and post the statistics in Table 1 to demonstrate the commonness of these error types.
Analysis and Possible Explanation We aim to analyze the cause of these errors via inspecting the inner workings of LLMs. We depict Logit Lens results of the examples of Table 1 (compositional queries) and their references (corresponding second-hop queries) in Figure 2, Leveraging Eqn. 1. Note that in Figure 2, results of second-hop inputs (subfigure (e)∼(h)) align well with the results in Figure 3. However, when we set our sights on results of compositional inputs (subfigure (a)∼(d)), we get clues about the above three error types. In (a, Distortion) we observe that the peak for o1 does not emerge at all (probability∼ 1|V | ), implying the distortion of the predictive information for o1 by context. In (b, Incomplete Reasoning), though o1 emerge in middle layers, it is not intense enough (in comparison with (f)) to arise the final result o2. In Figure 11, we show another example where the
peak probability of o1 aligns well with the result of the reference and correctly predict o2. In (c, Hasty Answer I) we observe that o1 emerge at the last layer, which is too late to incorporate second-hop information to generate o2. In (d, Hasty Answer II) although o1 (association football) also emerges, the peak probability of o1 is much lower than its reference (h). For comparison, we plot the Logit Lens of “the home country (r2) of Giorgio Chinaglia (s1)” for “Italy” in Figure 10, which aligns with its corresponding compositional query well, advocating that LLMs predict through short-cut. In summary, all of these errors can be attributed to improperly generating implicit reasoning results. The implicit reasoning results either (1): do not notably emerge (Distortion) or (2): emerge but not intensely or timely enough to raise the explicit reasoning results(Incomplete Reasoning and Hasty Answer).

heading: 4 Analyzing the Inner Hidden States of LLMs for Compositional Reasoning
text: Providing that LLMs are capable to perform compositional step-by-step reasoning (Hou et al., 2023), we hypothesize that they generate the implicit reasoning result o1 (the notation is aligned with Section 2.2) in the process of compositional reasoning, before finally obtaining the explicit reasoning result o2. We inspect inner hidden states of LLMs via Logit Lens (Section 4.1) and observe that implicit reasoning results emerge in middle layers, implying that they may play a role in the compositional reasoning process (Section 4.1). To verify this hypothesis, we design an intervention experiment (Section 4.2) and demonstrate the emerging of o1 has causal effect on predicting o2 in the output layer (Section 4.2).

heading: 4.1 Inspecting hidden states of LLMs
text: Given an input of a compositional two-hop knowledge item (s1, r1, o1) ⊕ (s2, r2, o2), we denote hl, (l ∈ [1..L]) as the hidden states at the position of last input token and l-th layer. Leveraging Eqn. 1 we tokenize implicit result o1 and explicit o2 into tokens: Ri (implicit) and Re (explicit), and inspect the information about Ri and Re in hl: L(hl, Ri) and L(hl, Re). We present the inspecting results averaging over D with LLaMA-2-7B in Figure 3(a). We observe that (1) both L(Ri, hl) and L(Re, hl) reach a peak and then decline with the layer increasing; (2) the peak of L(Ri, hl) appears
at the earlier layer than L(Re, hl). Then we use the corresponding second-hop queries (s2, r2, o2) (s2 = o1) to repeat the inspecting experiment. The averaged result is depicted in Figure 3(b). We get the similar observations with the compositional two-hop queries, to some extent aligning their reasoning processes: both of the compositional query (implicitly containing o1) and the secondhop knowledge query (explicitly containing o1) generate o1 in hidden states of middle layers before generating o2.
The insights gleaned from the emergence of implicit results suggest a potential influence of them on compositional reasoning. In the subsequent analysis, we endeavor to elucidate how implicit reasoning results, embedded within the hidden states of intermediary layers, exert a causal impact on
the generation of explicit reasoning results.

heading: 4.2 Verifying the Hypothesis via Intervention
text: We recall the notations defined before. The tokenizations of o1 and o2 are Ri and Re; the hidden state of the last token at the l-th layer is hl. Accordingly, the probability distribution over the output vocabulary set V (with Eqn. 2) is pl = softmax(vl) = softmax(hl · Wu) ∈ R|V |. Our aim is to demonstrate how the information about o1 encoded in hidden states of middle layers plays a causal role in the prediction of o2. The technique of Intervention (Pearl, 2001; Li et al., 2023a) fits the objective, where we strategically intervene on these inner hidden states to eliminate the information related to o1 (through Logit Lens) and observe the resultant impact on predicting o2.
Intervention We define the intervention Il : hl → h∗l , where h∗l denotes the intervened hidden state. v∗l is the corresponding logits (through Logit Lens) of h∗l : v ∗ l = h ∗ l · Wu. Denoting that (before intervention) vmin = min 0≤j<|V |
{vl[j]}, we expect v∗l meets the following constraints:
v∗l [j] = { vmin, j ∈ Ri, vl[j], j ∈ [0..|V |)/Ri,
(3)
Which means, observing from Logit Lens, we eliminate the bias on o1 in h∗l in the computation graph and minimize the side effects on the rest tokens4. We solve the linear system v∗l = h ∗ l ·Wu
4More discussion please refer to Appendix D.1.
to get h∗l : h ∗ l = v ∗ l W T u (WuW T u ) −1 (in case that WuW T u is not full-rank, we use the Moore–Penrose inverse (Dresden, 1920) instead). In our implementation, we calculate the difference value for the purpose of numerical stability:
h∗l = hl + (v ∗ l − vl)W Tu (WuW Tu )−1. (4)
Effect We define the effect El of an intervention Il is the difference between probabilities of predicting o2 (tokenization: Re) at the output layer L before and after the intervention:
El = pL[Re]− pIlL [Re]. (5)
Ideally, we expect the intervention Il has the effect of decreasing the probability of predicting the explicit reasoning result o2 (i.e., El > 0).
Result The Intervention experiment results (averaged over D) are depicted in Figure 4. For each experiment group, we set a comparison group where we intervene on |Ri| tokens that are randomly sampled from V . Comparing experiment groups and comparison groups, we observe there exist apparent positive effects (El > 0) when intervening middle layers (for both LLaMA-2 and OpenAlpaca, positive effects appear in 15-th to 20-th layers) for experiment groups, suggesting that the information about o1 may be generated and utilized for generating o2 in these layers. Meanwhile, there is nearly no notable positive effect for comparison groups across all layers. The results verify our hypothesis that the information around implicit reasoning results in middle layers play a role in predicting explicit reasoning results.

heading: 5 Locating Important Modules
text: In previous analysis, we attribute compositional reasoning errors to improperly generating implicit reasoning results. In this section, we aim to investigate if there sparsely exist some “key” modules (i.e., MHSA or MLP)5 in LLMs that are responsible for properly generating implicit reasoning results in hidden states of middle layers.

heading: 5.1 Locating Methodology
text: In Section 3, we observe that if inspecting results of the compositional query and its corresponding second-hop query align well, the compositional reasoning process is usually in smooth going. Given this, combining the key idea in Causal Mediation Analysis (Meng et al., 2022; Stolfo et al., 2023), we propose the following locating method. (1) We run the LLM twice: once with the compositional query in Dgap in the length of T1 and once with its corresponding second-hop query in the length of T2. For the compositional pass, we denote the module outputs in the computation graph as {ηtl |η ∈ {a,m}, l ∈ [1..L], t ∈ [1..T1]} (a for MHSA, m for MLP, l indexing layers, t indexing tokens). For the second-hop pass, we denote the outputs as {η̂tl |η ∈ {a,m}, l ∈ [1..L], t ∈ [1..T2]}. (2) We replace a single module output of interest in the compositional pass computation graph with its counterpart in the second-hop pass computation graph. We focus on two token positions: the last subject token (which refers to (s1, r1) for compositional queries, e.g., “the sports associated with Giorgio Chinaglia”) and the last token6. We denote the original probability of predicting o2 as p(o2) and the probability after replacement as p(o2|η̂t∗l → ηtl ). (3): We define the effect of the replacement η̂t
∗ l → ηtl as p(o2|η̂t ∗ l → ηtl )− p(o2).

heading: 5.2 Insight
text: We depict the Average Indirect Effect (AIE) of replacements over modules, tokens, and layers in Figure 5. We observe that replacing the MHSA output at the position of (last-token, 18\19-th layer) has the largest effect on finally predicting the correct answer o2. Interestingly, this coincides with the intervention experiment results in Figure 4, implying that MHSA modules of these positions play an important role in properly accumulating and
5We introduce the LLM architecture in Appendix C.1 6These two positions have been demonstrated as most
informative for factual reasoning (Meng et al., 2022).
leveraging implicit reasoning results.

heading: 6 Patching Compositional Reasoning
text: Grounded on the empirical insights in Section 4 and Section 5, we are poised to introduce the CREME approach, designed to correct compositional reasoning failures via editing the parameters of MHSA at the located positions. We demonstrate its superiority through comparative analyses with two recent baselines for correcting compositional reasoning (Sakarvadia et al., 2023; Ghandeharioun et al., 2024) and a a widely recognized model editing baseline (Meng et al., 2022).
Specifically, our edit objective is the MHSA output matrix at the l-th layer W lO (for detailed description, please refer to Eqn. 7). Following Santurkar et al. (2021), we view W lO as a linear associative memory (Kohonen, 1972): W lO ∈ Rd×d operates as a key-value store for a set of vector keys K = [k1|k2|...] and corresponding vector values V = [v1|v2|...], by solving (W lO)TK = V .
For a given compositional query and its corresponding second-hop query, we run the LLM twice: once with the compositional query and once with the second-hop query. In the first pass with the compositional query, the input of W lO at the last token position is k∗ ∈ Rd×1; in the second pass with the corresponding second-hop query, the output of W lO at the last token position is v∗ ∈ Rd×1. We aim to edit W lO to Ŵ l O such that:
minimize ∥(Ŵ lO)TK − V ∥2F and (Ŵ lO)Tk∗ = v∗,
where the Frobenius norm guarantees consistent predictions on irrelevant queries while the constraint implements the edit as an insertion of (k∗, v∗) into the linear memory Ŵ lO. Following Meng et al. (2022), we derive a closed form solution: Ŵ lO = W l O + (C −1k∗)TΛT where C =
KKT is a constant to estimate the uncentered covariance of k (note that k is randomly sampled from Wikipedia to represent irrelevant queries) and Λ = (v∗ − (W lO)Tk∗)/(C−1k∗)Tk∗. Hopefully, the edited LLMs are able to properly generate implicit reasoning results at the located position and thus alleviate failures of compositional reasoning.
What is worthy noting is that applying CREME only requires a single case of (compositional query, referenced second-hop query) while the patching effect can generalize to many other related cases. We showcase the effect of CREME in Table 2 and quantitatively discuss this generalization effect in Section 6.2.

heading: 6.1 Dataset, Baseline and Evaluation Metric
text: Dataset The dataset Dedit we use for editing and evaluating LLMs is built based on the Dgap filtered in Section 3. For each example in Dedit, it has the following fields: (1) Original input Io is a cloze test form of the compositional two-hop query. Accordingly, we also have the correct answer (ground-truth) and the originally predicted wrong answer for Io: Ao and Ão, respectively7. In the experiment, we use Io and its corresponding second-hop query to edit the LLM. (2) Paraphrasing input Ip is a paraphrase of Io. Note that Ao and Ão are also applicable to Ip. (3) Generalization input Ig is a compositional two-hop query where its first-hop sub-knowledge is shared with Io while the second-hop sub-knowledge is different from Io. We denote the correct answer for Ig is Ag. (4) Irrelevant input Ii is a compositional two-hop query that is irrelevant to Io and does not share the final answer with Io. Detailed information about Dedit is available in Appendix B.
Baseline We choose two related works in the field of correcting compositional reasoning errors through manipulating the inner workings of LLMs: Memory Injection (Sakarvadia et al., 2023) and CoT-PatchScopes (Ghandeharioun et al., 2024) as our baselines. Memory Injection enhances the compositional reasoning through explicitly injecting the implicit reasoning result (so-called “memory”) into the hidden states in the residual stream. CoTPatchScopes corrects the compositional reasoning through mimicking the noted Chain-of-Thought (CoT) reasoning (Wei et al., 2022) to re-route forward computation. Besides, we also compare
7e.g.,for the fourth case in Table 1:Ao=England; Ão=Italy.
CREME with ROME (Meng et al., 2022), a stateof-the-art model editing method. Detailed implementations are available in Appendix D.
Evaluation Metric In order to comprehensively validate the effectiveness of CREME, we propose four evaluation metrics: Correction, Paraphrasing, Generalization and Specificity. Following Sakarvadia et al. (2023), all the metrics are formulated on the basis of Improvement Percentage (IP), which is calculated as IP(I, A) = pM∗ (A|I)−pM(A|I)pM(A|I) . This formula quantifies the enhancement in prediction probability of an answer A given an input query I , facilitated by the post-edit LLM M∗ in comparison to the pre-edit LLM M. Specificially, Correction quantifies IP(Io, Ao) (larger is better); Paraphrasing is IP(Ip, Ao) (larger is better); Generalization is IP(Ig, Ag) (larger is better) and Specificity is IP(Ii, Ao) (smaller is better). CoT-PatchScopes, due to its nature of input-dependent, only fits the Correction evaluation. We report the average results over Dedit in Section 6.2. Note that to handling the multiple tokens in the answer, we calculate the probabilities for all of the tokens in the predicted probability distribution and report the mean value.

heading: 6.2 Experiment Results
text: The main experiment results are shown in Table 3. For brevity, we omit ×100% for each IP value. We observe that CREME achieves better performance than baselines on all metrics, not only achieving notable improvement on Io (the query used for editing), but also effectively generalizing to Ip (paraphrased queries). Interestingly, editing with Io also improves (at most +366%) the compositional reasoning on Ig (only sharing first-hop knowledge with Io), demonstrating the effectiveness of CREME on generating proper implicit rea-
soning results in middle layers. Besides, the Specificity score of CREME is low, showing that the CREME does not aimlessly improve the probability of predicting Ao for irrelevant inputs Ii. In comparison, the Correction score of Memory Injection (+221% for LLaMA-2) is almost the same with the original paper8 while we find it is less effective to generalize to Ip and Ig. Moreover, its high Specificity score implies its shortcoming of aimlessly improving the probability of predicting Ao. We also show IP(Io, Ão) in Figure 7. A good correction method should have little positive improvement on predicting the wrong answer Ão. We observe that p(Ão|Io) approximately remains unchanged with CREME, while is apparently enlarged with Memory Injection and PatchScopes.
One natural concern arises regarding the sufficiency of Correction and Paraphrasing metrics in practice. To this end, we evaluate the probability of an event where the probability of predicting Ao exceeds that of predicting Ão: p(Ao)>p(Ão). We compare CREME against baselines using this new metric and two types of input (Io and Ip) in Table 4. The results underscore CREME’s efficacy in significantly improving the event probability, thereby outperforming the unedited LLM and establishing a considerable lead over the two baselines.
Although CREME is not comparable to traditional model editing methods (the latter require Ao for editing, while CREME does not), we compare CREME with a well-regarded model editing method: ROME (Meng et al., 2022) for a comprehensive investigation. The results9 are shown
8Nonetheless, it still falls far behind CREME. Given that both CREME and Memory Injection aim to enhance the information of implicit reasoning results encoded in intermediary hidden states, we attribute the efficacy of CREME to its compatibility with models.
9Correction and Paraphrasing scores are using the event

heading: Evaluation Metrics C(↑) P(↑) G(↑) S(↓)
text: in Table 5. Our findings reveal that while ROME marginally surpasses CREME in terms of the Correction score of ROME – attributable to ROME’s direct application of Ao for editing and its optimization procedure designed to entirely fit p(Ao) – CREME performs obviously better than ROME in paraphrased, generalization and irrelevant cases. This highlights the effectiveness of CREME on correcting compositional reasoning.
To make readers have better sense of the realistic effect of using CREME to improve LLMs’ compositional reasoning performance, we also report the (1) decreasement percentage of log PPL values10 (of predicting correct answers) and (2) prediction accuracy in paraphrasing and generalization testing cases before and after applying CREME in Table 6.
In Figure 6, we show the effects of editing different layers, where results align well with the results of the locating experiment (Figure 5).
probability of p(Ao|I) > p(Ão|I). 10https://huggingface.co/docs/transformers/en/ perplexity

heading: 7 Related Work
text: Compositional Reasoning of LLMs LLMs fail to solve a large proportion of compositional multihop questions, even successfully solving all their single-hop sub-questions (Press et al., 2023; Dziri et al., 2023). Early works towards mitigating this issue typically prepend crafted demonstration exemplars containing the “thought process” of solving the compositional query step-by-step and encourage LLMs to imitate the process via in-context learning (Nye et al., 2021; Wei et al., 2022; Zhou et al., 2023; Drozdov et al., 2023; Press et al., 2023). Recent works turn to inspect the inherent compositional reasoning mechanism (Hou et al., 2023) of LLMs. Sakarvadia et al. (2023) manually injects implicit reasoning results into LLMs at the middle layers to correct compositional reasoning failures. (Ghandeharioun et al., 2024) fixes compositional reasoning errors through re-routing inner hidden representations in the computation graph to mimic chain-of-thought reasoning process. Nonetheless, their interventions in the reasoning process are rough so that the improvement is limited and hardly generalize to other related queries. To this end, we elaborately analyze the cause of compositional reasoning failures, locate a small set of parameters in LLMs that are responsible for such failures and precisely edit them to correct such failures. The work and Yang et al. (2024) are concurrent, where both of the two works show empirical evidence that LLMs can latently perform multi-hop reasoning internally.

heading: 8 Conclusion
text: In this paper we study and patch the compositional reasoning of LLMs. Through examining failure instances and conducting diverse analysis experiments, we demonstrate successful compositional reasoning within LLMs hinges on its awareness of generating and leveraging implicit reasoning results. Moreover, we locate few important MHSA modules in LLMs that are responsible for properly generating and leveraging implicit reasoning results via causal mediation analysis. To this end, we propose CREME, to compositional reasoning failures via editing the located MHSA parameters and empirically demonstrate its superiority.

heading: Limitations
text: Technique Part of our observation and experiments in Section 4 and Section 3 are on the basis
of Logit Lens (nostalgebraist, 2020). Though Logit Lens is a widely used tool for analyzing the inner workings of language models (Geva et al., 2022, 2023; Dar et al., 2023; Sakarvadia et al., 2023; Katz and Belinkov, 2023; Ram et al., 2023), we acknowledge that it is only an approximate way to interpret the information in the inner hidden states of the LLMs (Belrose et al., 2023). Nonetheless, the residual stream architecture of Transformers guarantees that Logit Lens makes sense to a large extent. In our experiments, we try to conduct experiments with different techniques for the crossvalidation of our observations and conclusions (By way of example, the observations in the locating experiments (Section 5) to some extent validate the observations of the intervention experiments in Section 4.2).
LLM Due to the constraints of available computation resource, we are able to conduct most of our experiments with LLMs of seven billion scale (LLaMA-2-7B (Touvron et al., 2023)) and three billion scale (OpenAlpaca-3B (Su et al., 2023b)). Both of these two LLMs are fully open-sourced and popular in academic community and real-world applications (Wu et al., 2023; Wang et al., 2024; Hou et al., 2023; Li et al., 2023b). In the future work, we aim to validate our conclusions on LLMs of larger scale.
Task In this work, we mainly focus on the task of the compositional reasoning on factual knowledge, which is generally pursued by lots of research works (Misra et al., 2023; Press et al., 2023; Zhong et al., 2023; Sakarvadia et al., 2023). We aim to validate our main conclusion about the significance of implicit reasoning results in the compositional reasoning process in other types of compositional reasoning task (Lu et al., 2023; Hou et al., 2023)(e.g., Arithmetic Reasoning for multiple operands) in the future work.

heading: Ethical Considerations
text: We study the inner workings for the compositional reasoning of LLMs, which helps the blackbox LLMs become more transparent and trustworthy (Räuker et al., 2023). The CREME method introduced in this work is originally designed for correcting the compositional reasoning failures of LLMs. CREME only require slightly update a small set of parameters in LLMs and can generalize to a number of related queries (paraphrased queries
or compositional queries sharing first-hop knowledge with the query used for conducting CREME). However, just like traditional model editing methods (De Cao et al., 2021; Mitchell et al., 2022; Meng et al., 2022, 2023; Bi et al., 2024), it may also be utilized to insert inaccurate (or out-of-date) information into the pretrained LLMs, potentially resulting in negative influences in real-word applications of LLMs such as information retrieval (Lian et al., 2020a) and recommendation (Lian et al., 2014, 2020b).

heading: Acknowledgements
text: Special thanks to Dr.Xiting Wang from Renmin University of China for her helpful comments to the early version of the work. We also sincerely thank the anonymous reviewers for their insightful suggestions to this work. The work was supported by grants from the National Key R&D Program of China (No.2021ZD0111801), the Research Grants Council of the Hong Kong SAR under Grant GRF 11217823 and Collaborative Research Fund C104223GF, the National Natural Science Foundation of China under Grant 62371411 and InnoHK initiative the Government of the HKSAR,Laboratory for AIPowered Financial Technologies.

heading: A Related Works on Mechanistic Interpretability and Model Editing
text: Mechanistic Interpretability and Model Editing Mechanistic Interpretability, interpreting inner workings of LLMs, is drawing an increasing attention of NLP researchers. Logit Lens (nostalgebraist, 2020) is proposed to interpret hidden states at the middle layers of LLMs via projecting them to the output vocabulary space with the LM head. Subsequent works (Geva et al., 2021, 2022;
Dar et al., 2023; Katz and Belinkov, 2023) further explain how LLMs build precise next token predictions. Another line of mechanistic interpretability works focus on inspecting factual knowledge encoded in the LLMs: they first locate such factual knowledge in pretrained LLMs (Dai et al., 2022; Geva et al., 2023; Li et al., 2023b) and then edit them through updating a small set of parameters of LLMs (Meng et al., 2022, 2023; Hase et al., 2023), which is so-called “locate-then-edit” model editing (Ju and Zhang, 2023). In this paper, we shed light on the mechanism of compositional reasoning on factual knowledge and borrow the idea from “locate-then-edit” model editing to locate the root and correct it to patch compositional reasoning failures of LLMs.

heading: B Datasets
text: Dataset for Non-Editing Experiments Here we mainly introduce the dataset we use for NonEditing experiments (including inspecting experiments in Section 4.1, intervention experiments in Section 4.2, inference experiments in Section 3 and locating experiments in Section 5.) The dataset D we use in this paper is sourced from (Zhong et al., 2023), a dataset containing plenty of highquality compositional multi-hop reasoning cases. For the ease of our study and following the setting of (Press et al., 2023), we collect 1,000 twohop knowledge items (each with its two single subknowledge) as the base of our dataset. For each datum in the dataset, it contains the following component: (1) four paraphrased compositional twohop knowledge (s1, r1, o1)⊕ (s2, r2, o2)(o1 = s2) queries: one of them is in Cloze-Test form and the other three is in Question form; (2) two paraphrased first-hop sub-knowledge (s1, r1, o1) queries: one is in Cloze-Test form and another is in Question form; (3) two paraphrased second-hop sub-knowledge (s2, r2, o2) queries: one is in Cloze-Test form and another is in Question form; and (4) the results for compositional reasoning: the intermediate implicit reasoning result o1 (meanwhile is the answer for the first-hop queries) and the final explicit reasoning result o2 (meanwhile is the answer for the second-hop queries). Following (Meng et al., 2022; Geva et al., 2023; Zhong et al., 2023; Press et al., 2023), We use the Question form queries in the inference experiment (Section 3) and ClozeTest form queries in most of the rest experiments in this paper. Below is an example for the datum in
our dataset.
{ "compositional question query": [
"Which writer's country of citizenship is the same as the author of \"Misery\"?", "What country does the author of \"Misery\" and another writer share their citizenship?", "What is the nationality of the author of \"Misery\"?"
], "compositional cloze query": "The
nationality of the author of \"Misery\" is",
"first-hop question query": "Who is the author of \"Misery\"?" "first-hop cloze query": "The author of \"Misery\" is" "second-hop question query": "What is the nationality of Stephen King?" "second-hop cloze query": "The nationality of Stephen King is", "compositional answer": "United States of America", // explicit reasoning result "first-hop answer": "Stephen King", // implicit reasoning result "second-hop answer": "United States of America"
}
Dataset for Editing Experiments Here we mainly introduce the dataset we use for conducting and evaluating CREME (Correcting Compositional Reasoning via Model Editing) in Section 6. The dataset Dedit we use for editing and evaluating LLMs is built on top of the dataset Dgap filtered in Section 3: for a LLM M: we focus on the example that M succeeds to predict the correct answer given any of single-hop inputs in it while fails to correctly predict the answer for the corresponding compositional two-hop input in it. In this section, we are going to correct these compositional reasoning failures. Specifically, for each example in Dedit, it has the following components: (1) Original input Io, refers to a cloze test form of the compositional two-hop knowledge mentioned above. Accordingly, we also have the correct answer (ground-truth) and the originally predicted wrong answer for Io: Ao and Ão, respectively11. (2) Paraphrasing input Ip, refers to a paraphrase (e.g., cloze test → question) of Io (we collect 3.39 Ip for each Io in average). Note that Ip shares the Ao and Ão with Io. (3) Generalization input Ig, refers to a verbalized compositional two-hop knowledge where the first-hop sub-knowledge is shared with Io and the second-hop sub-knowledge
11E.g., for the first case in Table 1: Ao=England; Ão=Italy.
is different from Io (we collect 2.64 Ig for each Io in average). We denote the correct answer for Ig is Ag. (4) Irrelevant input Ii, refers to a verbalized compositional two-hop knowledge that is irrelevant to Io and does not share the final answer with Io (we collect 9.49 Ii for each Io in average). Below is an example for the dataset (corresponding to the Incomplete Reasoning type of errors in Section 3).
{ "Original Input": "The capital of the
country that Lou Pearlman is a citizen of is",
"Correct Answer for I_o": "Washington, D.C.", "Predicted Wrong Answer for I_o": "United
States of America", "Paraphrasing Input":[
"What is the capital of the country to which Lou Pearlman belonged?", "Which city serves as the capital of the country where Lou Pearlman was a citizen?", "In which city is the capital of the country where Lou Pearlman had citizenship?", "The capital of the country to which Lou Pearlman belonged is",
... ], "Generalization Input": [
"The official language of the country that Lou Pearlman is a citizen of is", "What is the official language of the country that Lou Pearlman is a citizen of?",
... ], "Generalization Answer": [
"American English", "American English", ...
] "Irrelevant Input": [
"Which continent is the country that Emma Bunton is a citizen of located in?", "The official language of the country that Thierry Mugler is a citizen of is",
... ], "Irrelevant Answer": [
"Europe", "French", ...
] }

heading: C Language Models

heading: C.1 LLM Architecture
text: Current Large Language Models (LLMs, in this paper, we conduct most of the experiments with
two popular and open-sourced LLMs12: LLaMA-27B (Touvron et al., 2023) and OpenAlpaca-3B (Su et al., 2023b; Taori et al., 2023).) are mostly built on the basis of traditional Transformer (Vaswani et al., 2017) (Decoder). They are typically consist of an embedding layer E, an output language model (LM) head Wu and a stack of repetitive Transformer blocks between E and Wu.
Embedding Layer Given a tokenized input inp = [t1, t2, ..., tN ], where each ti (1 ≤ i ≤ N) is a one-hot vector of |V | (V is the vocabulary set) dimensions, the embedding layer is actually an embedding matrix E ∈ R|V |×d, projecting the input sparse one-hot vectors into d-dimensional hidden space: inp · E = [h10, h20, ..., hN0 ]. hi0 (1 ≤ i ≤ N) ∈ Rd is the initial hidden state that is forwarded into the first Transformer block (Note that we omit the description for the rotary positional embedding (RoPE) (Su et al., 2023a) added at each Transformer block of the network).
Transformer Block A Transformer block (or a Transformer layer) typically has two sub-modules: a Multi-Head Self-Attention (MHSA) layer and a Multi-Layer Perceptron (MLP) layer. We denote the hidden states at the input and output of the lth (1 ≤ l ≤ L) Transformer Block are hl−1 and hl respectively (Since hidden states of all token positions are forwarded parallelly, we define hl ≜ [h1l , h 2 l , ..., h N l ] ∈ RN×d to represent the whole hidden states of the l-th layer.). Then we have:
hl = hl−1 + al +ml ∈ RN×d (6)
where al and ml refer to the MHSA output and the MLP output.
MHSA layer of l-th Transformer block contains four matrices: W lQ,W l K ,W l V ,W l O ∈ Rd×d. Let H denote the number of attention heads. Then the parameters in each matrix can be equally divided into H parts: each of them is an individual attention head (e.g., for the j-th head, 1 ≤ j ≤ H): W l,jQ ,W l,j K ,W l,j V ∈ Rd× d H and W l,jO ∈ R d H ×d. Then we first compute the attention value for the j-th head: (M ∈ {0, 1}N×N is the attention mask
12Due to the page limit, we sometimes present the results with one of them while readers can find the rest results in Appendix E.
matrix)
Al,j = softmax( (hl−1W
l,j Q )(hl−1W l,j K ) T
√ d/H
⊙M)
headjl = A l,j(hl−1W l,j V ) ∈ RN×d/H
The final output of the MHSA al is to concatenate these heads together:
al = Concat(head1l , head 2 l , ..., head H l )W l O ∈ RN×d
(7)
MLP layer of l-th Transformer block contains two matrices: Wup ∈ Rd×d′ , Wdown ∈ Rd
′×d (in LLaMA-2 (Touvron et al., 2023), d′ = 83d) and a non-linear activation function SwiGLU (Shazeer, 2020) σ. The output of the MLP ml can be computed as follows:
ml = σ((al + hl−1)Wup)Wdown ∈ RN×d
LM Head Let us denote the output of the last Transformer block (at the position of last token) is hNL (for LLaMA-2-7B: L = 32; for OpenAlpaca3B: L = 26.). The LM head is a matrix Wu ∈ Rd×|V | to project the hidden state hNL ∈ Rd back to the output vocabulary space (probability distribution over the vocabulary set V ) to predict the next token:
pNL = softmax(h N LWu) (8)

heading: C.2 LLaMA-2
text: LLaMA-2 (Touvron et al., 2023) is a collection of pretrained and fine-tuned generative text models ranging in scale from 7 billion to 70 billion parameters. In this paper, due to the computation resource restraints, we focus on the 7 billion version: LLaMA-2-7b-hf13, which is a popular opensourced LLM in both academic researches and industrial applications. LLaMA-2-7B has 32 layers (32 transformer blocks), a vocabulary size of 32,000 and a hidden dimension of 4,096. In the inference experiments of this paper, we adopt the default generation configuration for LLaMA-2-7B provided by Meta:
\\LLaMA-2-7B generation configuration GEN_CONFIGS["llama2-7b"]={ "bos_token_id": 1, "do_sample": True,
13https://huggingface.co/meta-llama/ Llama-2-7b-hf
"eos_token_id": 2, "pad_token_id": 0, "temperature": 0.6, "max_length": 50, "top_p": 0.9, "transformers_version": "4.31.0.dev0"
}

heading: C.3 OpenAlpaca
text: OpenAlpaca (Su et al., 2023b) is also an popular instruction-following LLM14 (fully open-sourced version of Alpaca (Taori et al., 2023)). We adopt the 3 billion version: OpenAlpaca-3B15, for we want to introduce some variation of parameter scales into our experiments. OpenAlpaca-3B has 26 layers (26 transformer blocks), a vocabulary size of 32,000 and a hidden dimension of 4,096. In the inference experiments of this paper, we adopt the default generation configuration for OpenAlpaca3B provided by Su et al. (2023b):
\\OpenAlpaca generation configuration GEN_CONFIGS["openalpaca-3b"]={
"do_sample": True, "top_k": 50, "top_p": 0.9, "generate_len": 128
"transformers_version": "4.31.0.dev0" }
D Implementation Details
D.1 Intervention
In the Intervention experiments (Section 4.2), a natural worry about the preciseness of the “intervention” manipulation is whether our intervention will direct affect the probability (observing via Logit Lens) of explicit reasoning results or not. Hopefully, the intervention only works on the “implicit reasoning result” (Ri) while due to the restriction of softmax function, the explicit reasoning result might also be affected by the intervention. In practical, this effect (caused by softmax function) on the “explicit reasoning result” (Re) is rather insignificant (∼ 3e − 5) and always increasing the probability (given that the summation of all probabilities over the vocabulary is one, our intervention decrease the probability of Ri, naturally improving probabilities for all other tokens.), and hence we do not need to worry about this “side effect”. Another potential “side effect” brought by the intervention
14https://github.com/yxuansu/OpenAlpaca 15https://huggingface.co/openllmplayground/
openalpaca_3b_600bt_preview
is caused for the approximation when solving the inverse matrix with PyTorch 16. Sometimes, the numerical error brought by the approximation can slightly decrease the probability of Re (observing via Logit Lens at the intervened layer). To mitigate the possibility that the final effect (in Figure 4) is attributed to this “side effect”, We additionally apply the following re-checking procedure (Our aim is that (1): we re-check whether the intervention decrease the probability of Re, and (2): if so, we manually remedy this “side effect”.). We first calculate the intervened hidden state h∗l :
h∗l = hl + (h ∗ l − hl)
= hl + (v ∗ l − vl)W Tu (WuW Tu )−1
(9)
Concentrating on Re, we project h∗l to the raw logits h∗lWu and check if there is decreasement on the probability of Re.
∆vl[j] = { v[j]− (h∗lWu)[j], j ∈ Re 0, j ∈ [0..M)/Re
(10) Then we re-update the hidden state:
h∗,recheckl = h ∗ l +∆vlW T u (WuW T u ) −1 (11)
In Figure 4, the values is derived from the following procedures: (1) to make positive effect easier to observe, do clipping on the negative effects (v < 0 → v = 0) for each single datum; (2) do rescaling to the effects for each single datum: for effects of different layers, min-max scale them to [0, 1]; and (3) average the effects over the whole dataset.
D.2 Inference Experiment In line with Zhong et al. (2023) and Press et al. (2023), we adopt the question-form queries to check if LLMs have the single-hop knowledges and whether they can compose them together to answer compositional two-hop questions. The main reason behind using question-form queries is that it is convenient for us to use prompting and In-Context examples to make LLMs directly output the answer. As for the prompt for the question queries, following Zhong et al. (2023), we prepend eight different demonstrations (namely exemplars) to guide LLMs. Note that, in our experiments, we eliminate the possibility that LLMs directly “copy” the correct answer from the in-context demonstrations by
16https://pytorch.org/docs/stable/generated/ torch.linalg.inv.html
manually filtering out those demonstrations with the same answer with the questions we want to query. Below is an example for our prompting:
Q: In which country was Tohar Butbul granted citizenship? A: Israel\n // eight demonstrations Q: Who was Nissan 200SX created by? A: Nissan\n Q: What continent is the country where Prickly
Pear grows located in? A: Europe\n Q: In which country is the company that created
Nissan 200SX located? A: Japan\n Q: Which continent is the country where the
director of My House Husband: Ikaw Na! was educated located in? A: Asia\n Q: What country was the location of the Battle of Pressburg? A: Hungary\n Q: What is the country of citizenship of Charles II of Spain? A: Spain\n Q: Who was Chevrolet Biscayne created by? A: Chevrolet\n Q: What is the name of the head of state of the country that Ellie Kemper is a citizen of? //our query (e.g., compositional question)
D.3 Important Module Locating
We implement our locating method on the basis of Causal Tracing (Meng et al., 2022). Following Meng et al. (2022)’s implementation, we also use a “window” intervention(a few layers before and after the intervened layer). In their original codebase, they set window size to be 10. In our experiments: we find that setting window size to be 2 is enough for us to effectively locate important modules. The locating results shown in Figure 5 adopted the window size of 6. In Figure 5, the values is derived from the following procedures: (1) do re-scaling to the indirect effects for each single datum: map the original value (before replacement) to 0 and map the positive indirect effects to [0, 1] and negative indirect effects to [−1, 0] in a linear manner; and (2) average the indirect effects over the whole dataset.

heading: D.4 Model Editing:CREME
text: We implement our CREME on the basis of (hiyouga, 2023). The method is described in Section 6.
The edit objective is the MHSA output matrix of l-th layer W lO. W l O ∈ Rd×d operates as a key-value store for a set of vector keys K = [k1|k2|...] and corresponding vector values V = [v1|v2|...], by solving (W lO)
TK = V . For a given compositional two-hop query and its corresponding second-hop query, we run the LLM twice: once with the compositional query and once with the second-hop query. We denote that: in the first pass with compositional query, the input of W lO at the last token position
is k∗ ∈ Rd×1; in the second pass with the corresponding second-hop query, the output of W lO at the last token position is v∗ ∈ Rd×1. In practice, when calculating k∗ and v∗, we prepend tens of random tokens to the compositional query and the corresponding second-hop query to mimic context environments, and get multiple input vectors and output vectors. Then we average input vectors and output vectors of different context environment to get k∗ and v∗, respectively. The edited matrix is: Ŵ lO = W l O + (C −1k∗)TΛT where C = KKT is a constant to estimate the uncentered covariance of k (with a sample Wikipedia of text) and Λ = (v∗ − (W lO)Tk∗)/(C−1k∗)Tk∗. In our experiments, we also apply a “window” editing (edit a few consecutive layers). In the main experiments, we adopt the window size of 6 in accordance with the locating experiments.

heading: D.5 Memory Injection
text: We manually inject memories of implicit reasoning results in to the residual stream of middle layers. Note that in the original implementation (Sakarvadia et al., 2023), they set a hyper-parameter, magnitude, to control the strength of injection. In our experiments, we sweep over the possibilities of injecting memories into any single middle layer. For each layer, we search the magnitude from 1 to 10. As for the matrix used for projecting the implicit reasoning results from the vocabulary space back into the hidden space, we try three different approaches: W Tu (in line with the original paper), W+u (Moore–Penrose inverse) and W
T (WW T )−1. We find that W Tu is always more effective.

heading: D.6 CoT-PathScopes
text: We follow the original implementation in Appendix.E. of the PatchScopes paper (Ghandeharioun et al., 2024). Rerouting the hidden states (at the last token position) from source layers to target layers. We use the p ∗ M(Ao|Io)−pM(Ao|Io)
pM(Ao|Io) to select the best source layer and target layer.

heading: D.7 ROME
text: We adopt the hiyouga (2023)’s implementation of ROME (Meng et al., 2022). The hyperparameters is in line with their original implementation (hiyouga, 2023; Zhang et al., 2024):
layers=[5], fact_token="subject_last", v_num_grad_steps=20, v_lr=1e-1,
v_weight_decay=1e-3, clamp_norm_factor=4, kl_factor=0.0625,
Besides, following the convention of model editing works (Meng et al., 2022, 2023), we also use the Cloze-Test form queries to edit LLMs. Note that in compositional queries, the “subject” is usually expressed as the description text containing s1 and r1. We treat the description text as the “subject” (e.g., “The sport associated with Giorgio Chinaglia” (association football)).

heading: E Additional Results

heading: E.1 Logit Lens Inspecting Results
text: In this section, we mainly present (1): the statistical Logit Lens inspecting results, (2): a case validating our Hasty Answer II observation (in Section 3) and (3): a case validating our Incomplete Reasoning observation (in Section 3). The statistical inspecting result for OpenAlpaca-3B is depicted in Figure 9. Note that the emerging of “implicit result” seems not as notable as the results of LLaMA-2-7B in Figure 3, the reason is that the layers of emerging peaks for OpenAlpaca-3B are dispersive in the
middle layers. We also provide the Logit Lens inspecting results for a single case in Figure 8 for readers’ reference. The cases validating Hasty Answer II and Incomplete Reasoning are depicted in Figure 10 and Figure 11, respectively.
E.2 Intervention Results
We present the results for the Intervention experiment (in Section 4.2) in Figure 4. For each experiment group, we set a comparison group where we intervene on |Ri| tokens that are randomly sampled from V . Comparing experiment groups and comparison groups, we observe there exist apparent positive effects (El > 0) when intervening middle layers (for both LLaMA-2 and OpenAlpaca, positive effects appear in 15-th ∼ 20-th layers) for experiment groups, suggesting that the information about o1 may be generated and utilized for generat-
ing o2 in these layers. Meanwhile, there is nearly no notable positive effect for comparison groups across all layers. The results verify our hypothesis that the information around implicit reasoning results in middle layers play a role in predicting explicit reasoning results.

heading: E.3 Memory Injection
text: The heatmap of averaged results for Memory Injection (Sakarvadia et al., 2023) are depicted in Figure 12. According to this heatmap, for LLaMA2-7B: we adopt the magnitude of 7 and inject layer of 3, for OpenAlpaca-3B: we adopt the magnitude of 10 and inject layer of 26.

heading: E.4 PatchScopes
text: The heatmap for PatchScopes (Ghandeharioun et al., 2024) are depicted in Figure 13. The qualitative are basically in align with the original paper: positive effects distributed in the area where the source layer is larger than the target layer. According to this heatmap, for LLaMA-2-7B: we set the source layer to be 12 and the target layer to be 4, for OpenAlpaca-3B: we set the source layer to be 13 and the target layer to be 7.

heading: E.5 Additional Results of CREME
text: We additionally show IP(Io, Ão) in Figure 7. Hopefully, a good correction method has little positive improvement for the prediction of wrong answer Ão. We observe that p(Ão|Io) approximately remains unchanged for CREME, while is apparently enlarged with Memory Injection and PatchScopes. In Figure 6, we show the effects of different editing layer, where the effect of editing layer 19 largely surpasses editing other layers (5,10,23,29). This results align well with the results of the locating experiment (Figure 5).

heading: E.6 Showcase of CREME
text: We use specific cases to show the effect of leveraging CREME to correct the compositional reasoning failures of LLaMA-2-7B in Table 2.
(a) (b)
(c) (d)
Figure 13: PatchScopes Results: (a) LLaMA-2-7B, p ∗ M(Ao|Io)−pM(Ao|Io) pM(Ao|Io) ; (b) LLaMA-2-7B,
p∗M(Ão|Io)−pM(Ão|Io) pM(Ão|Io) ;
(c) OpenAlpaca-3B, p ∗ M(Ao|Io)−pM(Ao|Io)
pM(Ao|Io) ; (d) OpenAlpaca-7B, p∗M(Ão|Io)−pM(Ão|Io) pM(Ão|Io) .

heading: Future Work