(2024)

Abstract: We study whether Large Language Models (LLMs) latently perform multi-hop reasoning with complex prompts such as “The mother of the singer of ‘Superstition’ is”. We look for evidence of a latent reasoning pathway where an LLM (1) latently identifies “the singer of ‘Superstition”’ as Stevie Wonder, the bridge entity, and (2) uses its knowledge of Stevie Wonder’s mother to complete the prompt. We analyze these two hops individually and consider their co-occurrence as indicative of latent multi-hop reasoning. For the first hop, we test if changing the prompt to indirectly mention the bridge entity instead of any other entity increases the LLM’s internal recall of the bridge entity. For the second hop, we test if increasing this recall causes the LLM to better utilize what it knows about the bridge entity. We find strong evidence of latent multi-hop reasoning for the prompts of certain relation types, with the reasoning pathway used in more than 80% of the prompts. However, the utilization is highly contextual, varying across different types of prompts. Also, on average, the evidence for the second hop and the full multi-hop traversal is rather moderate and only substantial for the first hop. Moreover, we find a clear scaling trend with increasing model size for the first hop of reasoning but not for the second hop. Our experimental findings suggest potential challenges and opportunities for future development and applications of LLMs.1

heading: 1 Introduction
text: Recent works have shown that Transformerbased (Vaswani et al., 2017) Large Language Models (LLMs) store and retrieve factual information in their parameters to complete simple prompts such as “The mother of Stevie Wonder is” (Petroni et al., 2019; Meng et al., 2022; Geva et al., 2021,
*Corresponding authors. 1Our code and dataset are publicly available at https://
github.com/google-deepmind/latent-multi-hop-reasoning
1
2022, 2023; Allen-Zhu and Li, 2023a). In addition, LLMs have demonstrated remarkable in-context reasoning abilities when the necessary information is explicitly given as part of the input (Wei et al., 2022b). For example, models can infer “Lula” as a possible completion of “The mother of Stevie Wonder is Lula. The singer of ‘Superstition’ is Stevie Wonder. The mother of the singer of ‘Superstition’ is”. These findings raise a question: Do LLMs retrieve factual information stored in their parameters and perform latent multi-hop reasoning when the information to reason from is not given as a part of the input? For instance, when LLMs process the two-hop prompt “The mother of the singer of ‘Superstition’ is”, do they (1) figure out that “the singer of ‘Superstition”’ refers to Stevie Wonder and (2) use their knowledge of who Stevie Wonder’s mother is to complete the prompt?
Answering this question is important. Evidence for such latent multi-hop reasoning would suggest that the LLM can connect and traverse through implicit knowledge stored in their parameters rather
10210
than only storing information redundantly in its parameters. Future work could strengthen such paths of traversal, ultimately leading to more parameterefficient and controllable models. Conversely, a lack of evidence would indicate more fundamental limitations of the Transformer architecture or training. It would also have critical implications for model editing: if complex facts are recalled instead of inferred, editing only base facts will never be enough since the changes cannot propagate (Onoe et al., 2023; Zhong et al., 2023; Cohen et al., 2023).
In this work, we limit ourselves to prompts that express a composition of two facts such as “The mother of the singer of ‘Superstition’ is” that humans can complete with two hops by (1) inferring a bridge entity (e.g., Stevie Wonder) and (2) inferring an attribute of that entity (e.g., who his mother is). Then, we study how often LLMs process the prompt using a similar latent two-hop reasoning pathway, although this pathway may not be the most salient pathway that largely determines the predicted output. To this end, we first study these hops individually, as shown in Figure 1. To study the first hop, we propose the entity recall score to approximate LLM’s internal recall of the bridge entity by projecting specific hidden representations to vocabulary space. We test how changes to the input prompt affect this score. To study the second hop, we propose to measure the consistency score between the distributions for completions of the two-hop prompt and an equivalent recall-based one-hop prompt (e.g., “The mother of Stevie Wonder is”). We check how often an intervention to increase the entity recall score increases consistency as an indication of second-hop utilization. Finally, we investigate how frequently both steps coincide.
To study latent two-hop reasoning with diverse types of fact composition, we introduce TWOHOPFACT dataset, which is based on Wikidata (Vrandečić and Krötzsch, 2014) and consists of 45,595 two-hop prompts of 52 types of fact composition. We experiment with LLaMA-2 (Touvron et al., 2023) 7B, 13B, and 70B. Our findings can be summarized as follows. Across a wide range of fact composition types for the two-hop prompts, we find substantial evidence for the first hop of the multihop reasoning. In about 70% of the times where we change the prompt to indirectly mention the bridge entity, the later layers of the transformer show increased bridge entity recall. For the second hop and overall traversal, the evidence appears weaker: in
60% of the cases where we increase entity recall score, consistency goes up. Likewise, in about 40% of the time, both hops work together (compared to a random 25% baseline); changing the descriptive mention increases the entity recall score, and increasing this recall score increases consistency.
While the above aggregate statistics do not suggest a very prevalent use of the latent multi-hop reasoning pathway, it is worth pointing out that up to 23% of the fact composition types demonstrate strong evidence of latent multi-hop reasoning, occurring in more than 80% of the cases. This suggests that the pathway exists but is highly contextual. Additionally, we focus on a very narrow interpretation of the pathway – in reality, we expect it to be more distributed across layers and tokens. Hence, the effects we see might be a lower bound on the model’s ability to perform latent twohop reasoning. We also find striking scaling behavior: while the first hop clearly improves substantially with parameter count, the second hop (and the round-trip performance) remains relatively constant. This might indicate a fundamental limitation in today’s architecture or pretraining.
Our contributions can be summarized as follows: • We establish a framework for the investigation of latent multi-hop reasoning in LLMs and show its existential evidence.
• We construct the TWOHOPFACT dataset which consists of 45,595 two/one-hop prompts of 52 fact composition types, created using various types of entities and relations and diverse templates (§4).
• We propose two novel metrics, internal entity recall score and consistency score, as proxies of the degree of the LLM’s recall of an entity for its descriptive mention (§5.1) and the degree of the LLM’s utilization of its knowledge about the bridge entity’s attribute (§6), respectively.
• We propose a mechanism to investigate a latent reasoning pathway even when it is not the most salient pathway determining the prediction, by measuring the relative frequency of the expected causal effects (§6.2).

heading: 2 Related Works
text: Recent works have shown that LLMs demonstrate remarkable in-context reasoning ability via prompting, which scales with model size (Brown et al., 2020; Wei et al., 2022a,b; Zhou et al., 2022). On the contrary, when the information to reason from
is not explicitly given as part of the input, LLMs often fail to correctly perform multi-hop reasoning even when they know the answer to the single-hop sub-step (Ofir Press et al., 2023; Dziri et al., 2023). While there have been wide investigations on how in-context reasoning works (Chan et al., 2022; Akyürek et al., 2023; Dai et al., 2023; Von Oswald et al., 2023; Prystawski and Goodman, 2023; Feng and Steinhardt, 2024), such an investigation has not been actively done to understand how latent multi-hop reasoning works.
While there have been works to investigate latent reasoning of LLMs, the exploration has been mostly done with simple single-hop reasoning tasks (Meng et al., 2022; Geva et al., 2023; Chanin et al., 2023; Hernandez et al., 2024) and/or controlled lightweight training/finetuning (Allen-Zhu and Li, 2023a,b; Saparov et al., 2023; Berglund et al., 2023, 2024). Also, many of the works that aim to identify latent reasoning pathways or circuits, have focused on finding the most salient reasoning pathway for simple synthetic tasks and/or toy models (Nanda et al., 2022; Olsson et al., 2022; Brinkmann et al., 2023; Wang et al., 2023; Conmy et al., 2023; Hou et al., 2023; Lieberum et al., 2023; McGrath et al., 2023). On the other hand, we study the existence of a latent multi-hop reasoning pathway, which may not be the most salient, in pretrained LLMs without further training, using diverse types of natural two-hop prompts.
Model editing examines ways to amend factual knowledge in LMs (De Cao et al., 2021; Mitchell et al., 2022; Meng et al., 2022; Zhang et al., 2024). However, recent works have shown that the existing editing approaches, largely focusing on single fact edits, fail to propagate the edits to facts that depend on the edited fact (Onoe et al., 2023; Zhong et al., 2023; Cohen et al., 2023). Our work explores the possibilities that such propagation could work. Moreover, our work investigates a pathway that affects the consistency at inference, whereas
prior work in consistency has focused on quantifying inconsistency and improving consistency posthoc (Ribeiro et al., 2019; Li et al., 2019; Asai and Hajishirzi, 2020; Elazar et al., 2021; Kassner et al., 2021, 2023; Jang et al., 2023). Sakarvadia et al. (2023) aim to improve multi-hop reasoning accuracy with a hypothesis that the errors stem from failure to recall the latent hop, while we investigate the foundations of this hypothesis of whether the model actually performs such a latent multi-hop reasoning. Li et al. (2024) is a concurrent work showing that a large portion of multi-hop reasoning failure cases can be attributed to incorrectly performing or utilizing the first hop of the latent multi-hop reasoning.

heading: 3 Problem Formulation

heading: 3.1 Preliminaries
text: We consider facts, such as “The mother of Stevie Wonder is Lula”, as triplets (e, r, e′) of a subject entity e (e.g., Superstition), a relation r (e.g., mother), and an object entity e′ (e.g., Lula). Specifically, in our analysis, we focus on triplets where e′ is the only or the most well-known object entity for the relation r for e (e.g. the only mother of Stevie Wonder is Lula), and view r as a function e′ = r(e), where r(e) is the function expression and e′ is the value of the expression. We analyze how LLMs process the composition of two facts with a bridge entity e2 connecting them, ((e1, r1, e2), (e2, r2, e3)), of which the composition is represented as r2(r1(e1)). An example is shown in Table 1.
To query LLMs, we use a template τ(·) to convert expressions r2(e2) or r2(r1(e1)) into a prompt that can be completed correctly by the value of the given expression. For instance, the single-hop expression mother(Stevie Wonder) could be converted by τ(mother(Stevie Wonder)) to the prompt “The mother of Stevie Wonder is”, which can be correctly completed with “Lula”. Similarly, the two-hop expression
mother(singer(Superstition)) could be phrased by τ(mother(singer(Superstition))) as “The mother of the singer of ‘Superstition’ is” with the same correct completion. While τ(r2(e2)) and τ(r2(r1(e1))) have the same answer (“Lula”), the latter requires recalling two facts rather than one. Therefore, we call τ(r2(e2)) a one-hop prompt and τ(r2(r1(e1))) a two-hop prompt, and denote them as τ1H and τ2H, respectively.
We assume that the two-hop prompts yielded by τ(·) for r2(r1(e1)) always contain a noun phrase description of the bridge entity e2 using e1 and r1, e.g., “the singer of ‘Superstition”’ for Stevie Wonder. We denote this description as µ(r1(e1))) and call it the descriptive mention of the bridge entity e2.
Last, we denote the type of the fact composition of a two-hop prompt as “type(r2) of type(e1)’s type(r1)”, where “type(e1)’s type(r1)” represents the type of the bridge entity’s descriptive mention in the prompt. For example, the fact composition type of τ(mother(singer(Superstition))) would be “mother of song’s singer”.

heading: 3.2 Latent Multi-Hop Reasoning in LLMs
text: Humans possess the deductive reasoning ability to infer conclusions from given premises, such as deducing that r2(r1(e1)) = e3 given a premise stating that r1(e1) = e2 and another premise stating that r2(e2) = e3. This multi-hop reasoning (Welbl et al., 2018; Yang et al., 2018) involves identifying the bridge entity (e.g., that “the singer of ‘Superstition”’ is Stevie Wonder) and using it to solve for the final answer (e.g., that Stevie Wonder’s mother is Lula).
Our research explores the extent to which a pretrained Transformer-based Large Language Model (LLM) can perform similar multi-hop reasoning when completing a two-hop prompt. Given the complex nature of LLMs, which function through highdimensional and distributed representations, it’s unlikely for a single deterministic algorithm to govern their predictions except for under highly controlled and constrained setup (Nanda et al., 2022; Wang et al., 2023). Instead, LLMs may use aggregations from multiple inference pathways, ranging from shallow n-gram co-occurrence-based matching to deeper rule-based reasoning or even multi-hop reasoning, to make a prediction.
Therefore, to identify a pathway indicative of latent multi-hop reasoning, we focus on the internal dynamics of LLMs in processing two-hop
prompts rather than the most salient pathway that contributes the most to the output. This involves analyzing how the LLM’s recall and utilization of the knowledge r1(e1) and r2(e2) changes in response to certain alterations made while the LLM is processing a two-hop prompt, in what we consider as the first and second hop of reasoning, respectively.
Specifically, we investigate the following two key research questions (RQs): RQ1. How often does an LLM perform the first hop of reasoning while processing two-hop prompts? We view the first-hop reasoning as the LLM’s recall of the bridge entity for its descriptive mention. Therefore, we examine the frequency with which the LLM’s internal recall of the bridge entity increases when it encounters a descriptive mention of the bridge entity within a prompt. For instance, we investigate whether altering the prompt from “The mother of the singer of ’Thriller’ is” to “The mother of the singer of ’Superstition’ is” increases the LLM’s internal recall of Stevie Wonder.
RQ2. How often does an LLM perform the second hop of reasoning while processing two-hop prompts? We view the second-hop reasoning as the LLM’s utilization of the first-hop reasoning for the second hop. Therefore, we examine the frequency with which enhancing the LLM’s recall of the bridge entity for its descriptive mention improves its use of the knowledge about the bridge entity to answer the two-hop prompt. For example, we investigate if increasing the internal recall of Stevie Wonder for “the singer of ‘Superstition’” makes the LLM better utilize its knowledge of Stevie Wonder’s mother to complete the prompt.
By addressing these questions, we aim to identify evidence of LLMs leveraging a latent pathway for multi-hop reasoning.

heading: 4 TWOHOPFACT Dataset
text: To answer our questions with prompts of diverse fact composition types, we construct TWOHOPFACT using well-known named entities in Wikidata (Vrandečić and Krötzsch, 2014) and manually selected relations (Appendix A). TWOHOPFACT consists of 45,595 unique pairs of one-hop and two-hop prompts of 52 fact composition types constructed from the same number of fact triplet pairs ((e1, r1, e2), (e2, r2, e3)) as in Table 1. Appendix Table 3 shows example two-hop prompts for each fact composition type, and Appendix B provides detailed data statistics.

heading: 5 First Hop of Multi-Hop Reasoning
text: In this section, we answer RQ1 of how often an LLM performs the first hop of reasoning while processing two-hop prompts. We first introduce ENTREC as a metric to approximate the LLM’s internal recall of the bridge entity upon its descriptive mention in a prompt (§5.1). Next, we propose to measure how often this recall increases when changing the input prompt to indirectly mention the bridge entity (§5.2). Then, we evaluate this using TWOHOPFACT and answer RQ1 (§5.3).

heading: 5.1 Internal Entity Recall Score
text: We define ENTREC as a metric to measure the LLM’s recall of the bridge entity e2 within a twohop prompt τ2H. This is defined with respect to the hidden representation in a certain layer l, at the last position of the bridge entity’s descriptive mention in the two-hop prompt. This hidden representation is projected to the vocabulary space to calculate the log probability of the first token of the entity’s name (e.g., the first token of “Stevie Wonder”). Formally, let e(0)2 be the first token of e2, then:
ENTRECl(e2, τ2H) (1)
= log softmax(LayerNorm(xl)WU )index(e(0)2 ) ,
where xl ∈ Rh is the output from the l-th Transformer layer at the last token of the bridge entity’s descriptive mention in the two-hop prompt τ2H, and index(e(0)2 ) ∈ [0, V − 1] is the index of the token e(0)2 in the unembedding matrix WU ∈ Rh×V . LayerNorm is the layer normalization used for the last layer output xL−1 before projecting it to the unembedding matrix to obtain the output next-token probability distribution. Applying this normalization makes ENTRECL−1(e2, τ2H) compatible with the output probability of e(0)2 as the next token of the prefix of τ2H ending at the descriptive mention (e.g., “The mother of the singer of ‘Superstition”’).2 We interpret higher ENTRECl(e2, τ2H) as stronger internal recall of the bridge entity e2 at the l-th layer.
The proposed definition of ENTREC is inspired by previous works which report that the representation constructed at the last token position of a subject often plays an important role in encoding information about the subject (Meng et al., 2022; Geva et al., 2023), the work of nostalgebraist (2020)
2We omit the bias term as it often models the frequency of the token (Kobayashi et al., 2023), which we do not want to consider for measuring the internal recall of an entity.
that projects early-layer outputs to the vocabulary space, and the work of Geva et al. (2022) which shows that such projections at the last subject token position of one-hop prompts provide interpretable top-rank attributes that are semantically relevant to the subject. Although ENTREC assesses the recall of an entity with respect to only the first token of its name, it is directly related to how autoregressive LLMs process the input text and prepare the next token to generate. A control experiment in Appendix C validates ENTREC as a reasonable proxy for measuring the internal entity recall.

heading: 5.2 Experiment
text: Given ENTREC, we answer RQ1 by measuring how often the internal recall of e2 improves at layer l when modifying a two-hop prompt from τ ′2H to τ2H, where τ ′2H does not contain the descriptive mention of e2 while τ2H does. To be specific, we measure the relative frequency of τ2H in TWOHOPFACT where ENTRECl(e2, τ2H) > ENTRECl(e2, τ ′2H).
To construct τ ′2H, we alter the descriptive mention of the bridge entity in τ2H in two ways: by replacing e1 with e′1 such that µ(r1(e ′ 1)) does not point to e2, or r1 with r′1 to ensure µ(r ′ 1(e1)) does not refer to e2. Examples include substituting “the singer of ‘Superstition”’ in τ2H to “the singer of
‘Thriller’” or “a plagiarist of ‘Superstition”’. These adjustments are termed entity substitution and relation substitution, respectively.
For each two-hop prompt τ2H in TWOHOPFACT, we randomly select one e′1 from the same fact composition type and one r′1 from a set of predefined candidate relations (provided in Appendix Table 5) to create τ ′2H. We then measure the relative frequency of cases where replacing τ ′2H with τ2H via entity or relation substitution increases the recall of e2. A relative frequency above 0.5 suggests the LLM’s chance to perform first-hop reasoning exceeds the random chance for these prompts.

heading: 5.3 Results
text: There is substantial evidence of the first hop of reasoning, which becomes stronger with increasing model size. Figure 2 shows the relative frequency of the cases that the entity recall at each layer increases with entity and relation substitution. LLaMA-2 7B entity substitution result (Figure 2a) shows that the evidence of first-hop reasoning becomes clearer with increasing layer depth, peaking at 0.71 in layer 31. Relation substitution exhibits a slightly noisier pattern with a peak at 0.63 in layer
7B 13B 70B
0
0.25
0.5
0.75
model size
r e la
t iv
e fr e q u e n c y
0.71 (L31) 0.72 (L39)
0.78 (L79)
(a) RQ1 entity substitution result (§5)
7B 13B 70B
0
0.25
0.5
0.75
model size
r e la
t iv
e fr e q u e n c y
0.63 (L20) 0.64 (L36)
0.76 (L75)
(b) RQ1 relation substitution result (§5)
7B 13B 70B
0
0.25
0.5
0.75
model size
r e la
t iv
e fr e q u e n c y
0.64 (L30) 0.65 (L14) 0.61
(L40)
(c) RQ2 result (§6)
7B 13B 70B
0
0.25
0.5
0.75
model size
r e la
t iv
e fr e q u e n c y
0.46 (L30) 0.43
(L35)
0.45 (L73)
(d) RQ1&2 entity substitution result (§7)
7B 13B 70B
0
0.25
0.5
0.75
model size
r e la
t iv
e fr e q u e n c y
0.38 (L30) 0.37 (L35)
0.43 (L74)
(e) RQ1&2 relation substitution result (§7)
Figure 3: Experimental results with increasing scale of LLaMA-2. Technical details for all experiments in our work can be found in Appendix E.
20 (Figure 2e). As model size increases from 7B to 13B and 70B, first-hop reasoning occurs more frequently for both entity substitution and relation substitution. For the former, the maximum relative frequency rises from 0.71 (7B) to 0.72 (13B) and 0.78 (70B) (Figure 3a). For the latter, it increases from 0.63 (7B) to 0.64 (13B) and 0.76 (70B) (Figure 3b).
Relatively strong evidence supports the firsthop reasoning in up to 73% of fact composition types. With LLaMA-2 7B-13B-70B, 18/25/34 and 21/27/38 out of 52 of fact composition types exhibit maximum relative frequencies exceeding 0.8 for entity and relation substitution, respectively. In addition, 11 out of 52 types demonstrate such strong first-hop reasoning evidence robustly across all model sizes and substitution types. For example, the maximum frequency of “president of anthem’s country” (“The country with the national anthem ‘Azat u ankakh Artsakh’ is led by president”) shows the maximum frequency of 0.97/0.92/1.0 (Figure 2d) and 0.87/0.87/0.89 (Figure 2h) with each model and substitution, respectively. Individual fact composition types exhibit diverse patterns of relative frequency across layers.

heading: 6 Second Hop of Multi-Hop Reasoning
text: In this section, we answer RQ2 of how often an LLM performs the second-hop reasoning while
processing two-hop prompts. We view the second hop of reasoning as the LLM’s utilization of what it knows about the bridge entity’s attribute (Stevie Wonder’s mother) to answer the two-hop prompt about the same attribute of the entity referred to by the descriptive mention (the singer of ‘Superstition”s mother). Therefore, when an LLM performs the second hop, we expect to see a connection between its recall of the bridge entity (i.e. resolving the first hop) and its similarity in responding to a two-hop prompt and a corresponding one-hop prompt about the bridge entity’s attribute, e.g., the two-hop prompt
“The mother of the singer of ‘Superstition’ is” and the one-hop prompt “The mother of Stevie Wonder is”. Namely, the more strongly the model recalls the bridge entity (e.g., Stevie Wonder) while processing the two-hop prompt, the more similar the completion of this prompt should be to the completion of the one-hop prompt. In the following, we describe our approach for testing how often such a causal connection exists between entity recall and the similarity in the prompt completions, which we refer to as consistency.

heading: 6.1 Consistency Score
text: We define CNSTSCORE to measure how consistently an LLM responds to the two-hop and onehop prompts. Let pτ2H ,pτ1H ∈ RV be the output
probability distributions for a two-hop prompt τ2H and the corresponding one-hop prompt τ1H, respectively. Denoting H(Q,P ) = −∑V−1i=0 Pi logQi as the cross-entropy between probability distributions P and Q, we define:
CNSTSCORE(τ2H, τ1H)
= −0.5H(pτ2H ,pτ1H)− 0.5H(pτ1H ,pτ2H). (2)
This score evaluates the similarity between the two probability distributions by computing and averaging their cross-entropy, ensuring symmetry in the evaluation. The symmetry from averaging mitigates sensitivity to the individual distribution’s entropy levels, aiming for equal treatment of divergences in both directions.
Note that we use consistency instead of two-hop prompt completion accuracy or the probability of the ground truth answer because the latter metrics are insufficient to capture the second-hop reasoning for the cases where the corresponding one-hop prompt completion is incorrect. In addition, these metrics inherit noise from the choice of the ground truth answer or the set of answer candidates. On the other hand, comparing the similarity of the output distributions is not affected by the choice of ground truth, and provides a way to capture the second-hop reasoning even when the ground truth answer is not in the top-1 generation of the one-hop prompt.
Also, we do not choose to compare the completion strings or their binary accuracy of the one/twohop prompts because these metrics cannot capture subtle consistency differences in the probability distribution. We choose cross-entropy rather than Kullback–Leibler or Jensen-Shannon divergence because the latter metrics contain an entropy term that is irrelevant to consistency, but can dominate the score, diluting the cross-entropy signal. Higher consistency scores indicate greater similarity between the output distributions. In Appendix D, we provide empirical evidence for the consistency score being a reasonable approximation of the utilization of the model’s knowledge about the bridge entity’s attribute.

heading: 6.2 Experiment
text: Given ENTREC and CNSTSCORE, we answer RQ2 by measuring how often increasing the recall of the bridge entity e2 at the l-th layer increases the LLM’s consistency in answering the two-hop prompt with respect to the one-hop prompt. In other words, we examine whether in-
creasing ENTRECl(e2, τ2H) leads to increasing CNSTSCORE(τ2H, τ1H).
We would have been able to use differential calculus to obtain the answer by calculating the direction of change if CNSTSCORE(τ2H, τ1H) were directly dependent on ENTRECl(e2, τ2H). However, there exists no direct functional dependency between the two values. Instead, we leverage the shared reliance of both metrics on xl for computation where l ∈ [0, L − 1),3 redefining them as ENTREC(xl) and CNSTSCORE(xl) relative to xl. This reparameterization allows us to change the question to: if ENTREC(xl) is increased by altering xl, does CNSTSCORE(xl) also increase?
To explore this, we adjust ENTREC(xl) in the direction of its steepest increase, represented by ∇xlENTREC(xl), and observe the impact on CNSTSCORE(xl) by modifying xl according to a magnitude of change α:
x̂l(α) = xl + α∇xlENTREC(xl).
Subsequently, we calculate CNSTSCORE(xl) using x̂l(α),4 which allows us to express it as a function CNSTSCORE(α) of α. Then, we examine its derivative, ddαCNSTSCORE(α) ∣∣ α=0
to understand the direction of change at the current value. A positive derivative indicates that an increase in ENTREC(xl) leads to an increase in CNSTSCORE(τ2H, τ1H), while a negative one suggests the opposite. By assessing the relative frequency of positive gradients among the two-hop prompts in TWOHOPFACT, we quantify how often the LLM performs the second hop of the reasoning, with frequencies above 0.5 suggesting that the LLM’s chance to perform the second-hop reasoning exceeds random chance for these prompts.

heading: 6.3 Results
text: There is moderate evidence of the second-hop reasoning, which does not become stronger with increasing model size. Figure 4 shows the relative frequency of the cases where increasing the bridge entity recall increases the consistency. In LLaMA-2 7B, the middle and late layers exhibit a relative frequency higher than 0.5 (random chance) with statistical significance, peaking at 0.64 in layer
3CNSTSCORE(τ2H, τ1H) utilizes pτ2H , which utilizes x l for its calculation. However, only xl where l = 0, · · · , L − 2 are used to calculate the attention outputs at layers l = 1, · · · , L− 1, respectively, to get pτ2H .
4We use activation patching (Wang et al., 2023) to implement the replacement of xl with x̂l(α).
0 10 20 30
0
0.25
0.5
0.75
layer
r e la t iv e fr e q u e n c y CnstScore↑
(a) LLaMA-2 7B
0 10 20 30
0
0.25
0.5
0.75
layer
r e la t iv e fr e q u e n c y CnstScore↑
(b) LLaMA-2 13B
0 10 20 30 40 50 60 70
0
0.25
0.5
0.75
layer
r e la t iv e fr e q u e n c y CnstScore↑
(c) LLaMA-2 70B
0 10 20 30
0
0.25
0.5
0.75
layer
r e la t iv e fr e q u e n c y CnstScore↑
(d) Random LLaMA-2 7B
0 10 20 30 40 50 60 70
0
0.25
0.5
0.75
layer
r e la t iv e fr e q u e n c y CnstScore↑
(e) 70B result of “stock exchange of game’s developer”
0 10 20 30 40 50 60 70
0
0.25
0.5
0.75
layer
r e la t iv e fr e q u e n c y CnstScore↑
(f) 70B result of “mother of song’s singer”
0 10 20 30 40 50 60 70
0
0.25
0.5
0.75
layer
r e la t iv e fr e q u e n c y CnstScore↑
(g) 70B result of “founder of person’s undergrad university”
0 10 20 30 40 50 60 70
0
0.25
0.5
0.75
layer
r e la t iv e fr e q u e n c y CnstScore↑
(h) 70B result of “president of anthem’s country”
Figure 4: Relative frequency that stronger recall of the bridge entity at the l-th layer increases the consistency of the LLM. Bars are colored blue if the relative frequency is greater than or equal to 0.5 and red otherwise. We manually set the value of 0.5 at the last layer because the intervention does not affect the consistency at that layer.
30. Test result with a randomly initialized model verifies 0.5 as the randomness baseline (Figure 4d).
However, unlike the first-hop reasoning (§5), the second-hop reasoning does not strengthen with increasing model size; when scaling from 7B to 13B and 70B, the maximum relative frequency remains relatively stable at 0.64 (7B), 0.65 (13B), and 0.61 (70B), as shown in Figure 3c. This observation does not change even when the test is conducted using the log probability of the ground truth answer instead of CNSTSCORE (Appendix F). It is worth noting that this finding aligns with the observation of Ofir Press et al. (2023), that the single-hop question answering performance improves faster than the multi-hop performance as the model size increases, and thus the compositionality gap (the ratio of how often models can correctly answer all sub-problems but not generate the overall solution) does not decrease with increasing model size.
Relatively strong evidence supports the secondhop reasoning in up to 19% of fact composition types. With LLaMA-2 7B-13B-70B, 10/7/5 out of 52 of fact composition types exhibit maximum relative frequencies exceeding 0.8, respectively. Among them, “founder of person’s undergraduate university” and “president of anthem’s country” demonstrate such strong second-hop reasoning evidence across all model sizes, with a maximum frequency of 0.86/0.81/0.82 (Figure 4g) and 0.84/0.89/0.82 (Figure 4h), respectively.

heading: 7 Latent Multi-Hop Reasoning
text: In this section, we measure how often LLMs perform latent multi-hop reasoning while processing the two-hop prompt by combining our answers to RQ1 and RQ2. For each two-hop prompt, we consider successful outcomes for RQ1 (an entity recall
increase with entity/relation substitution) and RQ2 (a consistency increase with increased entity recall) as evidence of the first and second hops of reasoning, respectively. Four possible outcomes arise: (SS) success in both RQ1 and RQ2 that we view as the multi-hop reasoning; (FS) failure in RQ1 but success in RQ2; (SF) success in RQ1 but failure in RQ2; (FF) failure in both RQ1 and RQ2.
There is moderate evidence of the latent multi-hop reasoning, which sometimes becomes stronger with increasing model size. Figure 5 shows the relative frequency of the four cases, where green, blue, yellow, and red represent each of the cases of SS, FS, SF, and FF, respectively. LLaMA-2 7B exhibits a relative frequency for successful multi-hop reasoning (green) above random chance (0.25), peaking at 0.46 (entity substitution) and 0.38 (relation substitution). The likelihood of partial multi-hop reasoning (green + blue + yellow) exceeds 0.8 in later layers.
While entity substitution results do not show increased multi-hop reasoning with model size (Figure 3d), relation substitution exhibits a scaling trend. From 7B to 70B, the maximum relative frequency increases from 0.38 to 0.43, suggesting that larger models may facilitate multi-hop reasoning with relational changes (Figure 3e).
Relatively strong evidence supports latent multihop reasoning in up to 23% of fact composition types. Considering 0.82 = 0.64 as the threshold, with respect to LLaMA-2 7B-13B-70B, 7/3/12 types exceed the threshold with entity substitution and 3/3/9 types do so with relation substitution. The maximum frequency of “anthem of capital’s country” (“The national anthem of the country led by president Lazarus Chakwera is named”) exceeds
0 5 10 15 20 25 30
0
0.2
0.4
0.6
0.8
case EntRec↑ (RQ1 Success) & CnstScore↑ (RQ2 Success) EntRec↓ (RQ1 Failure) & CnstScore↑ (RQ2 Success)
EntRec↑ (RQ1 Success) & CnstScore↓ (RQ2 Failure) EntRec↓ (RQ1 Failure) & CnstScore↓ (RQ2 Failure)
r e la
t iv
e f
r e q u e n c y
0 10 20 30
0
0.25
0.5
0.75
layer
r e la t iv e fr e q u e n c y
(a) 7B entity substitution
0 10 20 30
0
0.25
0.5
0.75
layer
r e la t iv e fr e q u e n c y
(b) 13B entity substitution
0 10 20 30 40 50 60 70
0
0.25
0.5
0.75
layer
r e la t iv e fr e q u e n c y
(c) 70B entity substitution
0 10 20 30 40 50 60 70
0
0.25
0.5
0.75
layer
r e la t iv e fr e q u e n c y
(d) 70B entity substitution for “anthem of capital’s country”
0 10 20 30
0
0.25
0.5
0.75
layer
r e la t iv e fr e q u e n c y
(e) 7B relation substitution
0 10 20 30
0
0.25
0.5
0.75
layer
r e la t iv e fr e q u e n c y
(f) 13B relation substitution
0 10 20 30 40 50 60 70
0
0.25
0.5
0.75
layer
r e la t iv e fr e q u e n c y
(g) 70B relation substitution
0 10 20 30 40 50 60 70
0
0.25
0.5
0.75
layer
r e la t iv e fr e q u e n c y
(h) 70B relation subst. for “anthem of capital’s country”
this threshold across all models and substitutions with 0.68/0.82/0.66 (Figure 5d) and 0.74/0.82/0.68 (Figure 5h), respectively. Individual types show diverse patterns distinct from the overall dataset.

heading: 8 Discussion and Conclusion
text: Our work studies the latent multi-hop reasoning abilities of LLMs. We find strong evidence of latent multi-hop reasoning for certain fact composition types with the reasoning pathway utilized in more than 80% of the cases. However, the utilization is highly contextual; there are also fact composition types where we see weak or almost no evidence of reasoning. The evidence of second and multi-hop reasoning across the whole set of prompts is rather moderate and only substantial in the first hop.
Moreover, while we see a clear scaling trend with the first hop of the latent multi-hop reasoning pathway with increasing model size, we do not see such scaling evidence for the second-hop reasoning pathway. This could be the reason behind the observation of Ofir Press et al. (2023) that the compositionality gap (the ratio of how often models can correctly answer all sub-problems but not generate the overall solution) does not decrease with increasing model size.
Although our analysis is based on LLaMA-2 family of models of up to 70B parameters, our findings suggest potential limitations in the current scaling paradigm for promoting latent multi-hop reasoning. Thus, we may need to study the choice of pretraining data, loss functions that promote knowl-
edge retrieval and utilization, or model architectures with a stronger inductive bias towards internal knowledge representation for LLMs’ stronger latent reasoning abilities. However, analyzing the subset of prompts with strong evidence of multihop reasoning with respect to pretraining dynamics and data may give insights into the emergence of such abilities even in the context of the current pretraining and scaling paradigm.
Overall, our findings advance the understanding of LLM capabilities and can guide future research aiming to promote and strengthen latent multi-hop reasoning which is relevant for parameter efficiency, generalization, and controllability.

heading: 9 Limitations
text: Latent Multi-Hop Reasoning Pathway While we study one pathway for latent multi-hop reasoning (e.g., we test the use of the second hop by means of entity recall), considering the potential redundancy of inference pathways in LLMs (McGrath et al., 2023), other pathways might exist; the same information might be retrieved in different ways. Also, we don’t measure multi-hop reasoning end-to-end and track only the changes that occur in the first and the second hop with respect to a single layer, while the effect of the first hop of reasoning could possibly propagate to other layers. Hence, the effects we see might be a lower bound on the model’s ability to perform latent two-hop reasoning.
Dataset We aim to collect fact triplets (e, r, e′) such that e′ = r(e) is the only or the most famous object for the relation r for e. Although we use the entities with the most number of reference links and ensure that e′ is the only object entity at least among the collected fact triplets for this purpose, there are noises introduced from Wikidata. Besides, in reality, it is difficult to strictly satisfy the condition of “only” due to the vast amount of real-world knowledge that changes rapidly and dynamically.
Metrics Our measure of internal entity recall is an approximation as we use only the first token of the entity, although it is directly related to how LLMs process the input text and prepare the next token to generate. Moreover, the internal entity recall score is based on logit lens (nostalgebraist, 2020) which has shortcomings such as representation drift, bias, and brittleness (Belrose et al., 2023; Timkey and van Schijndel, 2021). However, these limitations have minimal effect on our analysis because our focus is not on making the prediction accurate in early layers as studied for adaptive computation methods such as early exit (Din et al., 2023), but to study the LLM’s internal dynamics as-is.

heading: Acknowledgements
text: We would like to thank Sang-Woo Lee, Jasmijn Bastings, William Cohen, and Owain Evans for the valuable feedback and discussions.

heading: A Dataset construction
text: We construct TWOHOPFACT using Wikidata (Vrandečić and Krötzsch, 2014) with the following data construction pipeline.
A.1 Data Selection We select relations and entities that are well-known and result in sufficient numbers of samples per relation. Relations are selected manually. At the time of querying Wikidata, we constrain entities to singular entities with natural language Wikipedia titles and select entities with a maximal number of reference links. We also exclude the cases of e1 = e2 that might allow trivial recall of e2 by directly copying from the input. In addition, we make sure that bridge entities e2 are unique among the facts of the same fact composition type to mitigate the imbalance in the bridge entity. Finally, we apply down-sampling to mitigate the imbalance in the fact composition type.
Relation Selection First, we determine the type of the bridge entity’s descriptive mention by selecting the type of entities e1 and relation r1 to collect r1(e1) = e2. The bridge entities we select have types like “song’s singer” (the singer of a specific song), “country’s anthem” (the country with a specific national anthem), “founder’s organization” (the organization founded by a specific person), and “organization’s ceo” (the CEO of a specific organization). For example, while there can be many authors for some novels, “author’s novel” is selected as a type of descriptive mention of the bridge entity because we can use only the novels with a single author. We determine 19 types of bridge entity’s descriptive mention with this process.
Now that we have “type(e1)’s type(r1)” determined, we determine the type of relations r2 to determine the type of the fact composition, “type(r2) of type(e1)’s type(r1)”. Note that “type(e1)’s type(r1)” determined in the previous step falls into the category of country, organization (organization, undergraduate university, game developer), real person (author, president, CEO, spouse, singer), fictional character (main character), movie, novel, or city (headquarters city). Note that “type(e1)’s type(r1)” is also the bridge entity itself that the descriptive mention refers to. Therefore, we select r2 that are likely to give us a sufficient number of (e2, r2, e3) where e3 is the only object entity satisfying the relation r2 for these categories of e2. As in the previous step, we select common relations
as r2. Using the selected types of r2, we create 52 fact composition types including “mother of song’s singer” (the city where the novel of a specific novel was born), “headquarterscity of video game’s developer” (the city where the headquarters of the developer of a specific video game is located), and “director of main character’s movie” (the director of the movie which has a specific character as the main character).
Querying Wikidata We collect the fact triplets of the selected fact composition types through Wikidata Query Service5 with one handcrafted query for each of the 52 fact composition types. When there are too many results for the API call to bring before a timeout occurs, we reduce the number of the results by filtering the results with the number of reference links and/or adding other conditions to the query.
For the relations that are subject to change by nature, e.g., CEO of a company, we retrieve the information at the time of January 1, 2022. We choose this timestamp considering the training time of LLaMA-2 (Touvron et al., 2023) models that we use for our study. To analyze LLMs trained at different times, filtering the dataset with model accuracy before the analysis or using the prompts of the fact composition types that are less likely to change over time (e.g., “founder of videogame’s developer”) would resolve potential issues from the temporality of the dataset.
A.2 Natural Language Templates
We manually create natural language templates. To this end, we first create descriptive mentions of the bridge entity. To create the descriptive mentions, we manually write r1-specific mention-constructing templates mr1(·). For example, msinger(·) = “the singer of ‘· · · ”’ creates µ(r1(e1))) = “the singer of ‘Superstition”’.
Next, we create one/two-hop prompt templates. We manually write r2-specific prompt-constructing templates tr2(·) that take a mention of the bridge entity e2 and form a prompt querying about e2’s relational attribute r2 in a way that the prompt can be correctly answered with a mention of e3. For example, tmother(·) = “The mother of · · · is” is used to create the one-hop prompt “The mother of Stevie Wonder is” and also the two-hop prompt “The mother of the singer of ‘Superstition’ is”.
5https://query.wikidata.org
We write one representative template for each mr1 and tr2 in a way that two-hop prompts are natural. Some examples of how the templates are used to construct the prompts are shown in Table 2. Afterward, we translate the collected fact triplets to pairs of two-hop prompts and one-hop prompts using the manually written templates. To represent entities in a string, we use the title of the entity’s Wikidata page. We ensure that the generated prompts are grammatically correct. Table 3 shows the actual examples of the two-hop prompts and the bridge entity for each fact composition type.

heading: B Dataset Statistics
text: TWOHOPFACT consists of 45,595 unique pairs of fact triplets ((e1, r1, e2), (e2, r2, e3)) of 52 fact composition types, translated into 45,595 one/twohop prompts. Figure 6 shows the distribution of the fact composition types. The distribution of the fact composition type is relatively balanced, with the type that has the largest portion covering only 7.41% of the dataset (“birth city of novel’s author”).
Figure 7a shows the percentage of the majority bridge entity e2, i.e., e2 that is utilized the most to construct the one-hop prompt that corresponds to each two-hop prompt. The highest percentage of majority bridge entity among all fact composition types is only 15%, showing that the dataset is not biased as favorable towards certain e2. Figure 7b
shows the percentage of majority e3 that serve as the ground truth answer for the two-hop prompts. Table 3 shows the number of two-hop prompts for
each fact composition type with examples. We ensure that the number of prompts for a fact composition type exceeds at least 30 for statistically significant results.

heading: C Justification of Internal Entity Recall Score: Appositive Generation Experiment
text: Experiment We demonstrate that ENTREC is a reasonable approximation of the internal recall of the bridge entity with indirect evidence. Note that ENTRECl(e2, τ2H) is calculated not at the last token of τ2H but at the last token of the bridge entity’s descriptive mention, where it is grammatically natural to prepend a comma followed by the name of e2 (e.g., “The mother of the singer of ‘Superstition’, Stevie Wonder”). In the resulting string, grammatically µ(r1(e1))) becomes the antecedent and e2 becomes the appositive; an appositive is a noun phrase that follows another noun phrase in opposition to it and provides information that further identifies or defines it, and the antecedent is the noun phrase that the appositive describes. Then, if ENTRECl(e2, τ2H) reasonably approximates the internal recall of the bridge entity e2, it is expected
0 10 20 30
uguniv of novel's author
ugmajor of novel's author
stockexch of vdgame's dev
stockexch of founder's dev
stockexch of ceo's orgz
spouse of song's singer
spouse of orgz's ceo
spouse of novel's author
spouse of cntry's president
president of capital's cntry
president of anthem's cntry
origcntry of mainchar's movie
mother of song's singer
mother of person's spouse
mother of novel's author
hqcntry of vdgame's dev
hqcntry of person's uguniv
hqcntry of founder's univ
hqcntry of founder's dev
hqcntry of ceo's orgz
hqcity of vdgame's dev
hqcity of person's uguniv
hqcity of founder's univ
hqcity of founder's dev
hqcity of ceo's orgz
founder of vdgame's dev
founder of person's uguniv
founder of ceo's orgz
father of song's singer
father of person's spouse
father of orgz's ceo
father of novel's author
director of mainchar's movie
creator of novel's mainchar
cntry of univ's hqcity
cntry of person's birthcity
capital of president's cntry
capital of anthem's cntry
birthcntry of song's singer
birthcntry of person's spouse
birthcntry of orgz's ceo
birthcntry of novel's author
birthcntry of cntry's president
birthcity of song's singer
birthcity of person's spouse
birthcity of orgz's ceo
birthcity of novel's author
birthcity of cntry's president
author of mainchar's novel
anthem of president's cntry
anthem of capital's cntry
actor of movie's mainchar
percentage of majority bridge entity e 2
Lin Daiyu
British North Borneo United States of America
Pebble in the Sky
Emmanuel Macron
Elmore Leonard
Ran Guron
John Nuttall
Mylène Farmer
Emmanuel Macron
Elmore Leonard
Ran Guron
Anna Wood
Mylène Farmer
Mexico
Montenegro
Tokyo Istanbul
Chief Inspector Armand Gamache
The Aristocats
L. Frank Baum
Brenton Avery
Ian Harkes
Britney Spears
Smartisan
United States Military Academy
Capcom
VanMoof
Gainax
London School of Economics and Political Science
University of California, Berkeley
Konami VanMoof
Gainax
London School of Economics and Political Science University of California, Berkeley
Capcom
L. Frank Baum
Irina Eduardovna Lazareva
Britney Spears The Aristocats
Nicaragua
Bolivia
Emmanuel Macron
Stephen King
Sundar Pichai Neil Young
Argo Blockchain
Meta Platforms Capcom
Stephen King
Stephen King
(a) Percentage of majority bridge entity e2
0 2 0 4 0 6 0 8 0 1 0
0
1 2 0 1 4 0 1 6 0 1 8 0 2 0 0
uguniv of novel's author
ugmajor of novel's author
stockexch of vdgame's dev
stockexch of founder's dev
stockexch of ceo's orgz
spouse of song's singer
spouse of orgz's ceo
spouse of novel's author
spouse of cntry's president
president of capital's cntry
president of anthem's cntry
origcntry of mainchar's movie
mother of song's singer
mother of person's spouse
mother of novel's author
hqcntry of vdgame's dev
hqcntry of person's uguniv
hqcntry of founder's univ
hqcntry of founder's dev
hqcntry of ceo's orgz
hqcity of vdgame's dev
hqcity of person's uguniv
hqcity of founder's univ
hqcity of founder's dev
hqcity of ceo's orgz
founder of vdgame's dev
founder of person's uguniv
founder of ceo's orgz
father of song's singer
father of person's spouse
father of orgz's ceo
father of novel's author
director of mainchar's movie
creator of novel's mainchar
cntry of univ's hqcity
cntry of person's birthcity
capital of president's cntry
capital of anthem's cntry
birthcntry of song's singer
birthcntry of person's spouse
birthcntry of orgz's ceo
birthcntry of novel's author
birthcntry of cntry's president
birthcity of song's singer
birthcity of person's spouse
birthcity of orgz's ceo
birthcity of novel's author
birthcity of cntry's president
author of mainchar's novel
anthem of president's cntry
anthem of capital's cntry
actor of movie's mainchar
percentage of majority e 3
Tao Huimin İstiklâl Marşı The Star-Spangled Banner
Robert E. Howard Amiens London Paris Tokyo Pierrefonds France
United States of America
France United States of America
United States of America
Mexico City Podgorica
United States of America
United States of America
Robert E. Howard Wolfgang Reitherman Benjamin Ward Baum
Langer Avery
Vince McMahon
Joe Jackson
Government of Finland
Thomas Jefferson
Masaya Nakamura
London Tokyo London Cambridge
Tokyo
United Kingdom
United States of America
United States of America
United States of America
United States of America
Cynthia Ann Stanton Linda McMahon
Lynne Spears
United States of America
Emmanuel Macron Luis Arce Brigitte Macron Tabitha King Anjali Pichai Daryl Hannah
New York Stock Exchange Nasdaq
Tokyo Stock Exchange
English
University of Maine
(b) Percentage of majority e3
Figure 7: Percentage of the most frequent entities for each fact composition type of TWOHOPFACT. The expanded forms of the abbreviations used for the fact composition types are listed in Table 4.
that there will be at least some layers l where increasing ENTRECl(e2, τ2H) increases the relative frequency of the LLM to generate e(0)2 with a relative frequency higher than random chance. In other
words, we check the relative frequency of the cases where increasing the entity recall score at a layer increases the probability of the model to output e(0)2 as the next token of a comma following the prefix of τ2H ending at the descriptive mention (“The mother of the singer of ‘Superstition’,”). We calculate this relative frequency as described in Section 6.2 but using the probability instead of CNSTSCORE.
Result Figure 8 demonstrates that, in most of the mid-late layers, increasing the latent recall of the bridge entity when the LLM processes µ(r1(e1))) also increases the relative frequency of the LLM to output e(0)2 to generate the appositive of µ(r1(e1))) followed by a comma.6 The result indicates that
6For this analysis, we exclude the cases where the descriptive mention ends with one of the following: ?’, .’, !’, ,’, ”, )’, ”’, where appending a comma introduces changes in the tokenization results for LLaMA-2.
ENTREC at the n-th token has controllability of the token to be generated as the n+ 2-th token to make it more likely to be the first token of the appositive, serving as an indirect evidence that ENTRECl(e2, τ2H) is a reasonable proxy of the internal recall of the bridge entity.

heading: D Justification of Consistency Score: Comparative Experiment with Chain-of-Thought Cases
text: Experiment We demonstrate that the proposed definition of CNSTSCORE(τ2H, τ1H) is a reasonable proxy of the utilization of what the LLM knows about the bridge entity’s attribute – the latent recall of its answer to τ1H – with indirect evidence. If the information to reason with is given as part of the input, e.g., if the given prompt is “The singer of ‘Superstition’ is Stevie Wonder. The mother of Stevie Wonder is named Lula. The mother of the singer of ‘Superstition’ is”, the LLM would not need to internally perform the multi-hop reasoning to refer to what its output to the one-hop prompt “The mother of Stevie Wonder is” is, but just copy the answer from the input. Therefore, CNSTSCORE of such a case will be lower than the case where the LLM needs to internally figure out what its answer to the one-hop prompt given the hint of who the descriptive mention refers to, e.g., “The singer of ‘Superstition’ is Stevie Wonder. The mother of the singer of ‘Superstition’ is”. Therefore, to check whether this is the case, we compare CNSTSCORE computed with the several Chain-of-Thought (CoT) style prompts τ ′, i.e., CNSTSCORE(τ ′, τ1H).
Result Figure 9 shows the distribution of CNSTSCORE computed with different styles of prompts τ ′ as written in the y-axis. The red case is the consistency score of the two-hop prompt that we mainly study in our work, which requires full multi-hop reasoning. Because no information to reason from is given in the input, CNSTSCORE is significantly lower than the cases of other CoT-style prompts. The blue case is where what the descriptive mention refers to is given as the input, but what the LLM knows about the bridge entity’s attribute
needs to be internally recalled and referred to. The green cases are where the bridge entity’s attribute, i.e., the answer to the prompt, is explicitly given in the input, and thus, the LLM does not need to refer to its answer to the one-hop prompt. The result demonstrates that the mean of CNSTSCORE is higher for the blue cases where the model is forced to refer to its answer to the one-hop prompt than in the green cases where the model does not need to refer to the answer. The difference between the red and the blue cases would have come from the existence of the information of the descriptive mention’s identity in the input prompt, which would have helped the LLM to use the connection to refer to what it knows about the bridge entity.

heading: E Technical Details
text: We modify the codebase of Nanda and Bloom (2022) to run the experiments before refactoring. We use 1-8 40GB A100 GPUs for the experiments. All experiments run in less than 24 hours. We use the model weights from HuggingFace Transformers (Wolf et al., 2020) and use full precision for LLaMA-2 7B and 13B and half-precision for 70B. The SPARQL queries for querying Wikidata are written with the help of GPT-4 (OpenAI et al., 2023).

heading: F Accuracy-based Analysis for the Second Hop of Multi-Hop Reasoning
text: We perform consistency-based analysis instead of an accuracy-based analysis because solely relying on the answer correctness has limitations in answering our research questions, as explained in Section 6.1. For further analysis, we present accuracybased results in this section.
Using log probability of the ground truth answer instead of the consistency score does not affect our findings. We perform the RQ2 experiment described in Section 6.2 not with CNSTSCORE, but with the output log probability of the first token of the ground truth answer, e.g., “Lula” for the two-hop prompt “The mother of the singer
of ‘Superstition’ is”. We measure the relative frequency using the 17,231/45,595 two-hop prompts of which the corresponding one-hop prompts, e.g., “The mother of Stevie Wonder is”, are completed with the ground truth answer by all of the 7B, 13B, and 70B models.
Figure 10 shows the results, where the top row contains the relative frequency with CNSTSCORE and the bottom row contains the relative frequency with the log probability of the ground truth answer. When scaling from 7B to 13B and 70B, the maximum relative frequency with CNSTSCORE is 0.64 in layer 30 (7B), 0.65 in layer 14 (13B), and 0.62 in layer 40 (70B). The maximum relative frequency with the log probability of the ground truth answer is 0.60 in layer 30 (7B), 0.62 in layer 13 (13B), and 0.59 in layer 40 (70B). While the values are slightly lower for the log probability than those for the consistency score, the overall trends are alike. Also, as also observed in Section 6.3 with consistency, the second-hop reasoning with the log probability does
not strengthen with increasing model size.
Filtering based on the accuracy of the one-hop prompt does not affect our findings. We test whether the second hop of the latent multi-hop reasoning is stronger for the two-hop prompts of which the corresponding one-hop prompts are completed correctly with one of the ground truth answer candidates. For this analysis, we filter the two-hop prompts into two sets: those where the corresponding one-hop prompts are correctly completed for all model scales (τ1H correct) and those where the corresponding one-hop prompt is not completed with any of the ground truth answer candidates for all model scales (τ1H incorrect). Since the trend of the relative frequency significantly varies for different fact composition types (Figure 4), for fair comparison, we make the distribution of the fact composition types of the two sets become the same by sampling. This results in two sets of 9,734 twohop prompts with the same distribution of fact com-
position types. Figure 11 shows that the overall trends are similar for the two sets, τ1H correct (top row) and τ1H incorrect (bottom row), and that the maximum relative frequency of the two sets do not differ much across different model sizes. When scaling from 7B to 13B and 70B, the maximum relative frequency for the τ1H correct set is 0.62 in layer 30 (7B), 0.65 in layer 14 (13B), and 0.62 in layer 46 (70B). The maximum relative frequency for the τ1H incorrect set is 0.65 in layer 29 (7B), 0.65 in layer 20 (13B), and 0.61 in layer 46 (70B).

heading: Future Work
text: Overall, our findings advance the understanding of LLM capabilities and can guide future research aiming to promote and strengthen latent multi-hop reasoning which is relevant for parameter efficiency, generalization, and controllability