Distributional reasoning in LLMs: Parallel Reasoning Processes in Multi-hop Reasoning

## Abstract

Abstract Large language models (LLMs) have shown an impressive ability to perform tasks believed to require "thought processes”. When the model does not document an explicit thought process, it becomes difficult to understand the processes occurring within its hidden layers and to determine if these processes can be referred to as reasoning. We introduce a novel and interpretable analysis of internal multi-hop reasoning processes in LLMs. We demonstrate that the prediction process for compositional reasoning questions can be modeled using a simple linear transformation between two semantic category spaces. We show that during inference, the middle layers of the network generate highly interpretable embeddings that represent a set of potential intermediate answers for the multi-hop question. We use statistical analyses to show that a corresponding subset of tokens is activated in the model’s output, implying the existence of parallel reasoning paths. These observations hold true even when the model lacks the necessary knowledge to solve the task. Our findings can help uncover the strategies that LLMs use to solve reasoning tasks, offering insights into the types of thought processes that can emerge from artificial intelligence. Finally, we also discuss the implication of cognitive modeling of these results.

## 1 Introduction

The spread of activation theory in cognitive psychology suggests that ideas and concepts are stored in a network of interconnected nodes in the brain []. When one node is activated through perception, memory, or thought, it triggers a cascade that activates related nodes, facilitating processes like memory retrieval [] and association generation []. This theory has been instrumental in understanding how people recall information and connect different concepts, influencing cognitive research and practical applications like semantic search algorithms [, , ]. An alternative approach In cognitive psychology is the propositional approach []. It contrasts sharply with the associative approach by focusing on the logical structure and truth values of beliefs and judgments rather than mere connections between ideas. Propositional reasoning concerns how individuals assess, validate, and infer relationships between different propositions, considering their truthfulness and logical consistency. This method involves a more deliberate and conscious level of thought, requiring the cognitive system to engage in analysis and critical thinking. On the other hand, the associative approach operates on automatic processes, where thoughts and memories are triggered by simple connections or links between ideas without evaluating their truth value [, , , , , , ]. This results in a more instinctual and less reflective form of cognition, demonstrating how both approaches play distinct roles in human thought and understanding.

In the field of artificial intelligence, large language models (LLMs) have demonstrated a remarkable capability to complete tasks believed to require "thought processes" [, , ]. Originating from cognitive psychology, this notion of a thought process hinges on the ability to manipulate information in an abstract space, commonly referred to as working memory [, ]. For example, consider the question: "What is the first letter of the name of the color of a common banana?". Did you say to yourself or imagine the word "yellow" when trying to answer? The average response will be “yes”. The chain-of-thoughts (CoT) method [] has been the most recent success story for LLMs in solving tasks that require holding an intermediate state. This method involves LLMs noting subtask answers, eventually leading to the final answer. This approach resembles the propositional reasoning approach, and LLMs will likely adopt this strategy to generate human-like text.

However, Unlike the case of CoT, which encourages the model to mimic a human-like thought process, the training process underlying LLMs imposes no constraints on the internal process that generates the output. Thus, when not writing down an explicit thought process, the model could adopt various strategies to solve multi-hop tasks (Figure 1). This raises an important question: what strategy does the model use when applying the implicit approach? Recent studies have investigated the mechanisms that enable models to directly answer multi-hop questions (i.e., through a single token prediction). Yang et al. [] showed that during inference, the embeddings at the position of the bridge entity’s descriptive mention offer a higher probability for the intermediate result than prompts that do not refer to this entity. Li et al. [] investigated the root causes of failures in directly answering compositional questions. Their findings revealed that successful prompt examples showed an increased probability of intermediate results in the middle layers. Both studies demonstrated through interference experiments that modifying the embeddings to increase the probability of the intermediate answer also affected the final answer.

Figure: (a) Chain of attribute extractions
Refer to caption: extracted/5679422/images/chain.png

This work focuses on compositional two-hop questions. These can be formalized as a sequence of two attribute extractions (e.g., What is the first letter of the name of the color of a common banana?); the second extraction relies on information from the first (y from yellow). The main findings of this work suggest that the middle layers of LLMs not only represent the results of the first attribute extraction (i.e., yellow) but also this phenomenon is distributed over the range of possibilities (i.e., yellow, brown, green). We propose that the first attribute extraction creates a distribution of possible attributes while the second extraction operates on this distribution simultaneously (Figure 1d). This concept resembles the spread of activation theory.

Our proposal, which we refer to as a distributional reasoning, is demonstrated by showing that activations of potential final answers in the output layer can be approximated using a linear model which operates on the potential intermediate answers from the middle layers. We also show that, after the middle layer of the network, the inference process of compositional reasoning questions is characterized by highly verbal and interpretable hidden embeddings, which can be divided into two phases: (1) Increasing the activation of potential intermediate answers and (2) reducing the activation of intermediate answers while enhancing potential final answers (example in Figure 2a). The majority of this phase transition is handled by the feed-forward blocks (see Appendix A). Without testing direct causality, we demonstrate a high correlation between the distributions of intermediate answers and their corresponding final answers (example in Figure 2b). Lastly, we conduct two experiments that show that LLMs use the same reasoning process even when they hallucinate their answers.

Figure: (a) Activations by layer
Refer to caption: extracted/5679422/images/logits_banana.png

By demonstrating that LLMs utilize both associative-like activations and structured propositional reasoning in their operations, the paper not only sheds light on the internal workings of these models but also provides a computational model that mirrors these two major cognitive approaches. This helps bridge the understanding between human cognitive processes and artificial reasoning mechanisms, contributing valuable insights to the ongoing debate on how cognition can be modeled and replicated in machines.

Contributions:

- •
Statistical analysis demonstrating that the reasoning outcome of the model can be approximated by applying a simple linear transformation to a small and interpretable subset of logits from the middle layers.
- •
Novel and interpretable analysis of the multi-hop reasoning process that considers parallel and alternative reasoning paths.
- •
New dataset of fake items, allowing us to track the process of activation of intermediate states disconnecting the process from the content stored in LLMs.
- •
Computational framework demonstrating the role of associations in structured reasoning.

Section 2 discusses related work, which includes reasoning in LLMs and approaches for interpretability. Section 3 defines crucial notations, describes our model for distributional reasoning, and provides details about the dataset we used. Section 4 presents our experiments that demonstrate the phenomenon of distributional reasoning, along with the detailed results. Section 5 discusses the implications of our results, including potential future directions, and Section 6 presents the limitations of our work.

## 2 Related Work

Reasoning in LLMs.
An established line of work has attempted to assess and enhance the capability of LLMs to solve complex tasks [, ]. Most recent successes were achieved thanks to the methods of Chain-of-Thought (CoT) [], which involves LLMs noting subtask answers. Others addressed the ability of LLMs to manage the entire reasoning process in its hidden layers and answer in a single token prediction [, , ].

Interpretability of inference processes.
Many studies have attempted to interpret the internal processes occurring in LLMs during prediction [, , ]. This included identifying the roles of various modules in the model [, , , ] and developing methods for verbally describing how the output prediction is constructed [, , ]. This paper contributes to the collective effort to understand the prediction processes in LLMs.

## 3 Definitions

### 3.1 Notation

In line with the notation used by Press et al. [], every two-hop compositional reasoning question can be formulated using five variables: Subject - the initial topic the question is about; Q1 - the first hop question that extracts an attribute from the subject; A1 - the answer for Q1; Q2 - the second hop question that extracts an attribute from A1; A2 - the answer for Q2, which should be the final answer to the entire question. Table 1 presents a concrete example of this formulation. In addition, this paper will use several more notations as follows: Category - This refers to a semantic group of attribute names (e.g., colors, letters, cities, etc.). Representative Token - This is a single token from the model vocabulary associated with a specific word or expression (e.g., "US" for "The United States", "P" for "Pound", etc.). At times, the term representative token may be shortened to "token".

**Table 1: Compositional reasoning notation. To illustrate the notation, we use our running example.**
| Notation | Example |
| --- | --- |
| Question | What is the first letter of the name of the color of a common banana? |
| Subject | Banana |
| Q1 | Color of (Banana) |
| A1 | Yellow |
| Q2 | First letter of (Yellow) |
| A2 | Y |

To analyze the extent to which a term is represented in a single embedding vector, we can utilize the LM head. The LM head is the matrix that the model uses to project the output of the final layer into a vector in the vocabulary space. We will use the term activation of a word in a specific layer to refer to the result of activating the LM head on the output of this layer and selecting the index of the representative token of this word from the result. Formally:

$$ $activation_{l}(word)={(Wx^{l})}_{t}$ $$

Where $x^{l}$ is the normalized output of layer number $l$, $W$ is the LM head, and $t$ is the index of the representative token of the word. We use the terms activation or logit interchangeably.

Lastly, we use the term activation vector of category, denoted as $\vec{A1}$ or $\vec{A2}$, to refer to the activations of an entire category.

### 3.2 Linear approximation of distributional reasoning

We aim to define the two-hop reasoning process in two stages: (1) from a prompt to an activation vector of the intermediate answers category ($\vec{A1}$), and (2) a transformation from this activation vector to the final activation vector ($\vec{A2}$). Stage (1) is operated by a function that extracts potential attributes from a given subject. We hypothesize in this work that Stage (2) can be modeled using a linear transformation between the two category spaces. According to our formulation there is a matrix $Q2$ that, given a subject and a function $f_{Q1}$, can approximate the final vector $\vec{A2}$ as follows:

$$ ${\vec{A1}}\in{\mathbb{R}^{c_{1}}},\vec{A2}\in{\mathbb{R}^{c_{2}}},Q_{2}\in{ \mathbb{R}^{{c_{2}}\times{c_{1}}}}$ $$

$$ $\vec{A1}=f_{Q1}\left(subject\right)$ $$

$$ $\vec{A2}=Q_{2}\times\vec{A1}$ $$

The variables $c_{1}$ and $c_{2}$ represent the sizes of the semantic categories of the intermediate and final answers, respectively. Most importantly, the $Q_{2}$ matrix is invariant to the subject, as it is defined solely by the second-hop question.

### 3.3 Datasets

All experiments conducted in our study are based on the Compositional Celebrities dataset presented by Press et al. []. We use $6,547$ prompts divided into $14$ question types for our models and analyses. Each question pertains to an attribute of a celebrity’s birthplace. For the semantic category of $\vec{A1}$ we used all of the $117$ countries used as intermediate answers in the dataset. For each of the 14 question types, the semantic category of $\vec{A2}$ is defined by all the final answers associated with that type. Regarding the representative tokens, for each word or term, we generally use the first token capable of completing the input prompt with that term. Additionally, the prompts were modified so that the next likely token would directly answer the two-hop question (see Section B.1). This was done to ensure that the model will attempt to predict relevant tokens for our experiment (i.e., tokens from $\vec{A2}$). Full details can be found in our codebase, which we include as part of our supplementary material.

#### 3.3.1 Hallucinations dataset

We introduce a unique dataset, based on the Compositional Celebrities dataset. This dataset is distinctive because it contains two-hop questions that do not have correct answers. It was designed to encourage the model to "hallucinate" potential answers and perform manipulations on them. It divided into to sets: The first set contains 1400 questions in the same format of the questions in the Compositional Celebrities dataset, but all questions are regarding fictitious persons (see name list in Section B.2). The second set contains 3 question types: “What is the color of the favorite fruit of <name>? The name of the color is”, What is the first letter of the name of the favorite fruit of <name>? The first letter is” and “What is the first letter of the name of the favorite vegetable of <name>? The first letter is”. Full details can be found in our codebase, which we include as part of our supplementary material.

## 4 Experiments and Results

In this section we display our main results. All the experiments mentioned were conducted using the open-source LLMs Llama-2 [] with size 7B and 13B, Llama-3 [] with size 8B, and Mistral [] with size 7B.

### 4.1 Linear transformation between token categories

To test our hypothesis regarding the existence of the Q2 matrix (see Section 3.2), we construct a linear model for each of the $14$ question types, following the same steps. We begin by extracting the logits of $\vec{A1}$ from every layer during the inference process. For each layer, we attempt to predict the logits of $\vec{A2}$ in the final layer by using a linear regression model coupled with the k-fold method ($k=5$). We fitted a linear model for each of the 14 categories, predicting all $\vec{A2}$ logits simultaneously. We then calculated $R^{2}$ between the predictions and true values for each of the $\vec{A2}$ logits predictions. For each category, we calculated the mean $R^{2}$ by averaging the individual $R^{2}$ values for each $\vec{A2}$ logit. The reported $R^{2}$ per category is this computed mean. Figure 3a presents an example of one regression model results, and the mean $R^{2}$ across all categories is presented in Figure 3b. Detailed results by LLM and category are presented in Section C.1.

The results show that once two-thirds of the model depth is reached, the activations of $\vec{A1}$ can linearly predict the activations of $\vec{A2}$ in the final layer, with a mean of $R^{2}>0.5$ across various models and question types. We interpret this observation as evidence of the strong association that occurs in LLMs between intermediate and final results in compositional reasoning.

In the next step, we repeat the same modeling method with a minor modification. This time, we attempt to predict the $\vec{A2}$ logits in the final layer using the same $\vec{A2}$ logits from each of the other layers. The results suggest that, on average across all question types, the logits from the mid-layers of $\vec{A1}$ provide more information about $\vec{A2}$ than the logits of $\vec{A2}$ themselves (Figure 3b). This again, supports the role of the intermediate answers in the forming of the final answers generated by the LLMs.

Figure: (a) Model predictions for calling codes
Refer to caption: extracted/5679422/images/multivariate_regression_25.png

### 4.2 Interpretable representation of the intermediate category

We continue by examining the dynamics of the activations of $\vec{A1}$ and $\vec{A2}$. Our analysis indicates that after the middle layers of the network, there is an increase in the activation of multiple tokens from $\vec{A1}$ (Figure 4a). On average, the embeddings from the mid-layers assign a high probability to the most relevant token of $\vec{A1}$, sometimes even making it the most probable next token, even though this token is unsuitable for continuing a coherent sentence. In the subsequent layers, a phase transition occurs where the tokens of $\vec{A1}$ decrease as the tokens of $\vec{A2}$ increase, continuing this trend until the output is generated (Figure 4a).

Interestingly, there seems to be a connection between the activation patterns of the two categories in terms of the order of the activations (Figure 4b). The activation patterns of all tested LLMs are displayed in Section C.3. To investigate the relationship between the two activation patterns, we created a new vector, $\vec{S1}$, by sorting the logits of $\vec{A1}$ in decreasing order of their activation. We then created the following $\vec{S2}$ vector: for every index $i$ in $\vec{S1}$, the value of $S2_{i}$ corresponds to the activation of the $\vec{A2}$ logit of the correct final answer that matches the representative token of $S1_{i}$. For example, in the banana-color question, if the sorted $\vec{S1}$ contains the activations of $[yellow,brown,green]$, the respective $\vec{S2}$ will contain the activations of $[y,b,g]$. We calculated the average of $\vec{S1}$ and $\vec{S2}$ across the entire dataset (6547 prompts) and selected the top 10 logits from each vector. As a result, we obtained a vector representing the average of the top 10 logits for $\vec{A1}$ and another vector of $\vec{A2}$ logits that correspond to these top 10 A1 logits. To study the correlation between the activation patterns of $\vec{S1}$ and $\vec{S2}$, we calculated the Spearman correlation between them. The mean results are presented in Figure 4c, and category-level results are detailed in Section C.3. The results indicate that, on average, once two-thirds of the model depth is reached, the most activated logits of $\vec{A1}$ are arranged in a pattern closely related to the order of the $\vec{A2}$ logits in the output layer.

These observations are important in terms of interpretability. The increase in the activations of $\vec{A1}$ provides a lens to examine the process that led the model to its answer. This can assist in verifying the validity of thought processes and in explaining hallucinations when the response is incorrect. In addition, it raises questions about the causality of the process. returning to the banana question: if the model strongly associates the activation of yellow with y, one could argue that the activations are independent, and only exist because both tokens are attributes of banana. In contrast, if the model activates yellow, brown, green, and subsequently activates y, b, g in the same order, it becomes more challenging to argue that the activations are independent.

Figure: (a) Top 10 logits
Refer to caption: extracted/5679422/images/all_top_10_A1_A2.png

### 4.3 Hallucinations experiments

To further test our formulation and dissociate the operations of the Q2 matrix from the model’s knowledge about the subject, we created two datasets of compositional questions based on the compositional celebrities dataset. We conducted two different experiments designed to make the model answer questions beyond its knowledge. This method is useful for demonstrating that the model uses valid reasoning processes, regardless of whether it can provide a correct answer.

Fictitious subjects.
To test the consistency of Q2, we generated a list of 100 fictitious names (see Section B.2). We then expanded each of the 14 question types with 100 prompts related to these fictitious names. We used the new prompts to evaluate our models using the following method: Initially, we fitted a linear model with Ridge regularization on the original prompts from the dataset. Then, we attempted to predict the A2 activations of the new prompts without additional training. An example of such generalization result is presented in Figure 5a, and the mean of $R^{2}$ by layer is presented in Figure 5b. All other experimental results are detailed in Section C.2. Even though the predictions are less accurate, the statistical connections derived from the dataset remain informative, even for fictitious subjects (mean $R^{2}>0.3$). The results suggest that the reasoning process is independent of the model’s training data. Linear models trained on the original dataset were able to generalize to prompts about fictitious subjects. This indicates that the same reasoning process occurs within the model, regardless of the subject.

Figure: (a) Predictions for calling codes of fictitious names
Refer to caption: extracted/5679422/images/fake_A2_25.png

Fictitious attributes.
We used 1000 person names from the Compositional Celebrities dataset and generated new two-hop question types related to unusual attributes of the subjects (e.g., their favorite fruit, see Section 3.3.1). Assuming that information regarding favorite fruits is less likely to appear in the dataset, this allows us to test whether the reasoning process remains valid under out of distribution question domains. We repeated the same modeling method described in Section 4.1, and selected results are presented in Figure 6. All other experimental results are detailed in Section C.2. The results suggest that distributional reasoning process exists in out-of-distribution domains as well.

Figure: (a) Fruit color
Refer to caption: extracted/5679422/images/fruit_color_scatter_25.png

## 5 Discussion

This paper presents evidence of distributional reasoning in multi-hop question tasks, providing insights into the types of thought processes that can emerge from artificial intelligence. We demonstrated that by selecting a subset of tokens representing a semantic category of intermediate results, the tokens of potential final results can be approximated using a simple linear transformation. Our findings indicate that, on average, the intermediate results can explain at least 50% of the variance in the final activation results. Additionally, we demonstrated that during inference, the network’s middle layers activate a small subset of tokens representing potential intermediate answers. This subset corresponds to another small subset activated in the output layers, representing potential final answers. This observation implies the presence of parallel reasoning paths, which are highly interpretable. Finally, through two dedicated experiments, we demonstrated that LLMs can manipulate information in a valid reasoning process, even when the information is hallucinated.

The dynamic we capture, where the intermediate answers seem to be significant in the forming of the final answers, offers a novel cognitive approach for modeling together association and explicit reasoning. This bridges a gap that was observed by cognitive sciences decades ago and emphasis the role of AI research in cognitive modeling.

## 6 Limitations

There are a several limitations concerning the presented results. First, despite the variety in the types of questions we use, they have a similar general structure. Altering this structure could lead to different results. Second, it is worth noting that different prompt structures, question types, or subjects could lead the model to employ various solving strategies (see Figure 1). The statistical nature of the learning process likely encourages the model to utilize a variety of strategies and combine them when solving two-hop questions. Third, the proposed analysis cannot account for semantic categories that lack clear representative tokens, such as years. Future work will need to explore the mechanisms in these cases. Lastly, although the statistical analyses in this paper are quite convincing, they do not show direct causality. Future work will need to take this into account.

## Appendix

## Appendix A The reasoning process path

A previous study by Geva et al. [] investigated the information flow in attribute extraction prompts. Their findings indicated that a significant part of the process in the initial layers occurs at the position of the subject prompt. This stage of processing is referred to as "subject enrichment". Following this stage, as the authors reported, the information from this process propagates to the final index. The remaining process is primarily handled in the final index, leading up to the model’s output. Moreover, Li et al. [] identified critical modules for multi-hop reasoning tasks. They found that, up to the middle layers, the feed-forward blocks at the subject’s position were the most significant. In the later stages, the most important modules were the multi-head attention blocks and the feed-forward at the final index.

In order to verify these observation on our dataset, we conducted an interference experiment for each prompt in the dataset as follows: At first, we used the model to predict the most probable token after this prompt, and saved its probability as the $baseline$ probability. Next, for each layer of the model, we input the same data into the model but interfered the prediction process. We replaced the embeddings at all positions in that layer with zeros, except for the last index. After the inferred inference, we saved the updated probability of the token from the first round. For each layer $l$, we calculate its $intervention_score$ as follows:

$$ $intervention_score^{l}=1-\frac{prob}{baseline}$ $$

The average $intervention_score$ across the entire dataset is presented in Figure 7 (using Llama-2-13B model). The results show that on average, the influence of other token positions on the output probability significantly reduces after the 15th layer, reaching minimal effect from layer 25 onward. Considering our observations from Section 4.2, it appears that the increase in activation of the $\vec{A1}$ logits (as shown in the Figure 4a) corresponds to an information flow from other token positions. It also appears that the phase transition in the embeddings, where $\vec{A1}$ activations decrease as $\vec{A2}$ enhances, is managed solely at the last token index.

Figure: Figure 7: Intervention score using Llama-2-13B model. The average significance of any token index, except for the last one, dramatically decreases after the 15th layer.
Refer to caption: extracted/5679422/images/zero.png

## Appendix B Datasets

### B.1 Prompt modifications

To enhance the probability that the next predicted token will directly answer the two-hop question, we have added a suffix to each prompt in the compositional celebrities dataset. The specific suffixes for each category are outlined in Table 2.

**Table 2: Prompt modifications**
| Question type | Original prompt | Suffix | Comments |
| --- | --- | --- | --- |
| callingcode | What is the calling code of the birthplace of <name>? | The calling code is + |  |
| tld | What is the top-level domain of the birthplace of <name>? | The top-level domain is . |  |
| rounded_lng | What is the (rounded down) longitude of the birthplace of <name>? | The longitude is | ended with space or "-" depends on the country |
| rounded_lat | What is the (rounded down) latitude of the birthplace of <name>? | The latitude is | ended with space or "-" depends on the country |
| currency_short | What is the currency abbreviation in the birthplace of <name>? | The abbreviation is " |  |
| currency | What is the currency in the birthplace of <name>? | The currency name is " |  |
| ccn3 | What is the 3166-1 numeric code for the<br>birthplace of <name>? | The numeric code is | ended with space |
| capital | What is the capital of the birthplace of <name>? | The capital is |  |
| currency_symbol | What is the currency symbol in the birthplace of <name>? | The symbol is " |  |
| rus_common_name | What is the Russian name of the birthplace of <name>? | The common name in Russian is " |  |
| jpn_common_name | What is the Japanese name of the birthplace of <name>? | The common name in Japanese is " |  |
| urd_common_name | What is the Urdu name of the birthplace of <name>? | The common name in Urdu is " |  |
| spa_common_name | What is the Spanish name of the birthplace of <name>? | The common name in Spanish is " |  |
| est_common_name | What is the Estonian name of the birthplace of <name>? | The common name in Estonian is " |  |

### B.2 Fictitious names list

For the creation of our hallucinations dataset (see Section 3.3.1), we used Gemini [] for auto generating the following list of 100 fictitious names: Scarlett Evans, Oliver Morgan, Eleanor Clark, Finley Cooper, Violet Gray, Carter Edwards, Alice Brooks, Samuel Parker, Willow Moore, Henry Mitchell, Isla Bennett, Leo Turner, Evelyn Carter, Wyatt Peterson, Harper Garcia, Lucas Ramirez, Luna Patel, Logan Martin, Scarlett Lopez, Aiden Sanchez, Chloe Lee, Owen Perez, Riley Daniels, Liam Davis, Nora Robinson, Caleb Wright, Hazel Young, Elijah Thompson, Aurora Jones, Ryan Lewis, Zoey Walker, Dylan Baker, Penelope Harris, Gabriel Allen, Charlotte Campbell, Nicholas Taylor, Amelia Jackson, Jackson Moore, Evelyn Garcia, Matthew Ramirez, Luna Lopez, Benjamin Daniels, Maya Bennett, Alexander Turner, Ava Davis, Ethan Johnson, Riley Brooks, William Peterson, Aurora Sanchez, Noah Lewis, Zoey Baker, Dylan Harris, Penelope Allen, Gabriel Campbell, Charlotte Taylor, Nicholas Jackson, Amelia Moore, Jackson Garcia, Evelyn Ramirez, Matthew Lopez, Luna Daniels, Benjamin Bennett, Maya Turner, Alexander Davis, Ava Johnson, Ethan Brooks, Riley Peterson, William Sanchez, Aurora Lewis, Noah Baker, Zoey Harris, Dylan Allen, Penelope Campbell, Gabriel Taylor, Charlotte Jackson, Nicholas Moore, Amelia Garcia, Jackson Ramirez, Evelyn Lopez, Matthew Daniels, Luna Bennett, Benjamin Turner, Maya Davis, Alexander Johnson, Ava Brooks, Ethan Peterson, Riley Sanchez, William Lewis, Aurora Baker, Noah Harris, Zoey Allen, Dylan Campbell, Penelope Taylor, Gabriel Jackson, Charlotte Moore, Nicholas Garcia, Amelia Ramirez, Jackson Lopez, Evelyn Daniels, Matthew Bennett.

## Appendix C Semantic transformations experiments

All experiments in this study were conducted using a cluster service with servers that include a single GPU and 30GB RAM, or through [Google Colab](https://colab.research.google.com/) services on a T4 server. The experiments were conducted using the following large language models: Llama-2-13B , Llama-2-7B, Mistral-7B (with 8-bit quantization method), and Llama-3-8B.

### C.1 Main results

We fitted a linear model for each of the 14 categories, predicting all $\vec{A2}$ logits simultaneously. We then calculated $R^{2}$ between the predictions and true values for each of the $\vec{A2}$ logits predictions. For each category, we calculated the mean $R^{2}$ by averaging the individual $R^{2}$ values for each $\vec{A2}$ logit. The reported $R^{2}$ per category is this computed mean. Results for each category, at two-thirds of the model’s depth, can be found in Table 3. Average of Mean $R^{2}$ by layer can be found in Figure 8.

Figure: (a) Llama-2-13B
Refer to caption: extracted/5679422/images/llama13/all_correlation_layers_with_control.png

### C.2 Hallucinations experiments results

The results of the hallucinations experiments (see Section 4.3) are detailed by category and LLM in Table 3. The outcomes for the fictitious subjects experiments are shown under the FN columns, while the results for the fictitious attributes experiments appear in the bottom rows.

**Table 3: $R^{2}$ of linear regressions models. Columns A1 and A2 represent the categories of the semantic transformations predicted by the models; The results for the model at two-thirds depth of the LLM are displayed in the $\frac{2}{3}L$ columns; The FN columns show the results for the experiments involving fictitious subjects; The final divided layers correspond to the experiments with fictitious attributes.**
|  | Model |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Transformation | Llama2-13B | Llama2-7B | Mistral-7B | Llama3-8B |  |  |  |  |  |
| A1 | A2 | $\frac{2}{3}L$ | FN | $\frac{2}{3}L$ | FN | $\frac{2}{3}L$ | FN | $\frac{2}{3}L$ | FN |
| countries | calling codes | 0.86 | 0.61 | 0.76 | 0.47 | 0.84 | 0.68 | 0.56 | 0.4 |
| countries | domains | 0.72 | 0.42 | 0.58 | 0.45 | 0.6 | 0.41 | 0.59 | 0.29 |
| countries | longitudes | 0.54 | 0.27 | 0.61 | 0.36 | 0.67 | 0.34 | 0.54 | 0.19 |
| countries | latitudes | 0.78 | 0.37 | 0.54 | 0.18 | 0.68 | 0.46 | 0.57 | 0.26 |
| countries | currency shorts | 0.74 | 0.58 | 0.67 | 0.5 | 0.68 | 0.45 | 0.72 | 0.53 |
| countries | currency names | 0.75 | 0.47 | 0.69 | 0.45 | 0.72 | 0.44 | 0.69 | 0.46 |
| countries | iso 31661-1 | 0.52 | 0.32 | 0.64 | 0.1 | 0.5 | 0.42 | 0.58 | 0.29 |
| countries | capitals | 0.59 | 0.48 | 0.39 | 0.19 | 0.6 | 0.4 | 0.59 | 0.26 |
| countries | currency symbols | 0.78 | 0.53 | 0.68 | 0.4 | 0.71 | 0.5 | 0.78 | 0.58 |
| countries | russian names | 0.22 | 0.22 | 0.32 | 0.33 | 0.46 | 0.5 | 0.26 | 0.26 |
| countries | japanese names | 0.45 | 0.42 | 0.34 | 0.28 | 0.53 | 0.54 | 0.4 | 0.31 |
| countries | urdu names | 0.46 | 0.14 | 0.49 | 0.07 | 0.62 | 0.41 | 0.36 | 0.06 |
| countries | spanish names | 0.3 | 0.17 | 0.35 | 0.37 | 0.42 | 0.3 | 0.38 | 0.22 |
| countries | estonian names | 0.34 | 0.33 | 0.48 | 0.47 | 0.55 | 0.46 | 0.33 | 0.26 |
| fruits | colors | 0.45 |  | 0.52 |  | 0.33 |  | 0.39 |  |
| fruits | letters | 0.27 |  | 0.44 |  | 0.39 |  | 0.42 |  |
| vegetables | letters | 0.42 |  | 0.46 |  | 0.38 |  | 0.56 |  |

### C.3 Activation patterns

Figure 9 presents the activation patterns of the top 10 $\vec{A1}$ logits and their corresponding $\vec{A2}$ logits (see Section 4.2) of each LLM. Table 4 presents the Spearman correlations of the top 10 $\vec{A1}$ logits and their corresponding $\vec{A2}$ logits sorted by LLM and layer.

Figure: (a) Llama-2-13B
Refer to caption: extracted/5679422/images/llama13/all_top_10_A1_A2.png

**Table 4: Spearman correlation for average 10 top answers. The $\frac{1}{2}L$ and $\frac{2}{3}L$ columns correspond to the results at half and two-thirds of the model depth, respectively. Note: ${}^{***}p<0.001$, ${}^{**}p<0.01$, ${}^{*}p<0.05$.**
|  | Model |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Llama2-13B | Llama2-7B | Mistral-7B | Llama3-8B |  |  |  |  |
| Question type | $\frac{1}{2}L$ | $\frac{2}{3}L$ | $\frac{1}{2}L$ | $\frac{2}{3}L$ | $\frac{1}{2}L$ | $\frac{2}{3}L$ | $\frac{1}{2}L$ | $\frac{2}{3}L$ |
| Calling code | $0.98^{***}$ | $1.00^{***}$ | $0.38$ | $0.96^{***}$ | $0.72^{*}$ | $0.99^{***}$ | $0.89^{***}$ | $0.99^{***}$ |
| Domain | $1.00^{***}$ | $1.00^{***}$ | -$0.85^{**}$ | $0.99^{***}$ | -$0.72^{*}$ | $0.99^{***}$ | $0.76^{*}$ | $1.00^{***}$ |
| Longitude | $0.92^{***}$ | $0.92^{***}$ | $0.94^{***}$ | $0.95^{***}$ | $0.92^{***}$ | $0.92^{***}$ | $0.49$ | $0.58$ |
| latitude | $0.44$ | $0.44$ | $0.19$ | $0.24$ | $0.54$ | $0.54$ | $0.77^{**}$ | $0.77^{**}$ |
| Currency short | $0.95^{***}$ | $0.95^{***}$ | $0.15$ | $1.00^{***}$ | $0.64^{*}$ | $1.00^{***}$ | $0.72^{*}$ | $0.94^{***}$ |
| Currency name | $0.99^{***}$ | $0.99^{***}$ | -$0.32$ | $0.90^{***}$ | $0.5$ | $0.99^{***}$ | $0.47$ | $0.77^{**}$ |
| ISO 3166-1 | $0.1$ | $0.1$ | -$0.90^{***}$ | -$0.96^{***}$ | $0.89^{***}$ | $0.95^{***}$ | -$0.45$ | -$0.44$ |
| Capital | $0.96^{***}$ | $0.99^{***}$ | $0.2$ | $0.92^{***}$ | $0.93^{***}$ | $1.00^{***}$ | -$0.09$ | -$0.1$ |
| Currency Symbol | $0.99^{***}$ | $0.99^{***}$ | -$0.77^{**}$ | -$0.18$ | $0.12$ | $0.99^{***}$ | $0.72^{*}$ | $0.59$ |
| Russian name | $0.95^{***}$ | $0.95^{***}$ | $0.3$ | $0.98^{***}$ | $0.79^{**}$ | $0.95^{***}$ | -$0.68^{*}$ | -$0.68^{*}$ |
| Japanese name | $0.94^{***}$ | $0.94^{***}$ | $0.92^{***}$ | $0.96^{***}$ | $0.49$ | $0.70^{*}$ | $0.85^{**}$ | $0.88^{***}$ |
| Urdu name | $0.79^{**}$ | $0.81^{**}$ | -$0.05$ | -$0.28$ | $0.32$ | $0.35$ | -$0.81^{**}$ | -$0.48$ |
| Spanish name | $0.93^{***}$ | $0.93^{***}$ | $0.82^{**}$ | $0.95^{***}$ | $0.42$ | $0.61$ | $0.95^{***}$ | $0.96^{***}$ |
| Estonian name | $0.41$ | $0.44$ | $0.43$ | $0.96^{***}$ | -$0.62$ | $0.02$ | $0.33$ | $0.37$ |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: See the weakness.
soundness: 3
strengths: - This paper addresses a critical challenge in understanding LLM reasoning by probing the multi-hop reasoning processes. The discussion on human cognitive processes and LLM reasoning mechanisms is also inspiring.

- The study presents a clear illustration of distributional reasoning in LLMs (esp. with the example in Figure 2), which is a novel finding to my knowledge.
summary: This paper introduces a method to analyze multi-hop reasoning in LLMs by modeling their prediction process as a linear transformation between semantic spaces. The authors find that intermediate model layers create interpretable embeddings representing potential answers, activating specific token subsets and suggesting parallel reasoning paths. This framework offers insights into how LLMs mimic thought processes, even without direct task knowledge, with implications for understanding AI reasoning and cognitive modeling.
weaknesses: - While the illustration of distributional reasoning is interesting, the remaining analysis seems confusing to me. For example, what's the motivation of the analysis that uses the matrix to approximate the second-hop reasoning? The paper mentioned "This method reveals the extent to which the second-hop operation is invariant to the prompt specifics." why is this an important question to explore? and what's the takeaway from this analysis?

- The writing needs to be improved to make the logic clearer. Especially, there should be strong and clear motivation before going into any notations and technical details of the analysis. The current version makes readers very easy to get lost.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: **Q1** Does this analysis generalize to any sort of task that isn't 2-step compositional reasoning? What classes of task/problem does this framework make sense for, and what kind would it be unrealistic?

**Q2** This analysis suggests that the suggest-then-narrow-down reasoning approach can be __induced__ from the model's intermediates, but that is not the same thing as the model necessarily __using__ that approach. How can we have confidence that the model is actually using this reasoning chain and not rely upon other forms of reasoning instead/ in addition?
soundness: 3
strengths: * The paper is well-written and well-motivated. 
* The study design is clear and effective, and the results are interesting and well-supported. The secondary experiments (4.2 on the interpretability of the intermediate layers, and 4.3 on whether this type of reasoning still holds when the answer is hallucinatory) do a good job of preemptively answering follow-up questions I had on the original study. 
* The finding will be of interest to people concerned with whether current LLMs can shed light on questions in cognitive modeling of reasoning processes.
summary: This paper provides a study of the internal reasoning processes of LLMs when answering compositional questions. 
They find that one can apply linear models on the activations for specific task-related tokens at intermediate transformer layers to approximate the reasoning of the model at the "outcome" final prediction layer. This supports a hypothesis that LLMs consider a distribution of possible answers at intermediate layers then narrow down the final answer at later ones, in a process akin to the parts of the "spread of activation" and propositional reasoning theories of human cognition.

The authors consider a synthetic dataset (the Compositional Celebrities dataset from Press et al 2023, as well as carefully crafted variants to test for hallucinatory scenarios) and provide strong evidence for their hypothesis. The authors argue that transformer-based LLMs can be used to test questions in cognitive modeling.
weaknesses: **W1** The primary weakness of the paper is that it only considers one clean, synthetic dataset with one type of question. It is unclear whether such a clean result would hold on a more realistic reasoning dataset. The authors acknowledge this in their limitations section, but it is still the case that this paper shows __one__ scenario in which a clean form of reasoning can be approximated using linear modeling for __one__ kind of problem. The paper would be stronger with at least one other dataset that is more complex than 2-hop compositional questions and/or noisier than the templatic questions in Compositional Celebrities. Otherwise, this analysis should really be considered exploratory rather than comprehensive.

**W2** l501 "This bridges a gap that was observed by cognitive sciences decades ago and emphasis the role of AI
research in cognitive modeling." -- it would be helpful to describe more specifically how the result contributes the debate surrounding this gap (i.e., why they are incongruous) more specifically in the context of existing work. The paper seems to argue that both associative and propositional reasoning can take place at the same time in a single framework -- whether this is possible has been the subject of debate for many years, and various other cognitive models have been proposed that integrate both paradigms to differing degrees, e.g. [ACT-R](https://wires.onlinelibrary.wiley.com/doi/full/10.1002/wcs.1488?casa_token=9nMiql_9dRMAAAAA%3A1kKURVyse66RR4wdR3wxa1DDWQI_HO0D-oUAPfHepYuJzEhYj_0Jo77bV5qSxGHQq9O4Sz3VPpYa) and [CLARION](https://en.wikipedia.org/wiki/CLARION_(cognitive_architecture)). Comparing your consideration of transformers to these other frameworks and discussing how the contributions differ would benefit the paper.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: See the questions raised in "Weaknesses".
soundness: 2
strengths: 1. The paper proposes a hypothesis to interpret the multi-hop reasoning ability of LLM and designs sound experiments to test the proposed hypothesis. The findings provide interesting insights on neural cognition and reasoning.
2. The paper is clearly written. The experiments are well-designed with main results (Section 4.1 & 4.2) providing direct support to the hypothesis and additional results (Section 4.3) ruling out the confounding factor of the possibility of LLM simply memorizing knowledge.
summary: This paper investigates the mechanism of multi-hop reasoning of LLM. Specifically, the paper uses a compositional two-hop question answering setup as a test bed for the investigation. It proposes the hypothesis, referred to as distributional reasoning, that the first attribute extraction creates a distribution of possible attributes on which the second extraction can operate simultaneously to derive the final answer. To test the proposed hypothesis, the paper constructs a linear model to predict the activation of the final layer based on the activations of intermediate layers. The results indicate that once the model depth reaches a certain threshold, the activations of intermediate layers can linearly predict the activations of the final layer. Additionally, the paper demonstrates that the intermediate layers activate a small subset of tokens representing potential intermediate answers.
weaknesses: 1. While I can follow the proposed hypothesis and think the experiments are well-designed for testing them, I am unsure whether we can draw a conclusion that LLMs use distributional reasoning to answer multi-hop questions. As the authors also point out in the Limitations section, the paper only experiments on the two-hop question answering setup and all the questions are synthesized from one data source, the results may be different on questions with very different structures. For example, I am wondering whether the proposed hypothesis still hold on answering math problems that require multiple steps to solve. Or, still within the question answering setup, can the concept of distributional reasoning be identified on answering n-hop questions when n > 2?
2. Another major concern is the implication of the proposed hypothesis. Although the experimental results support the proposed hypothesis, it does not exclude other possibilities - therefore, I personally think the contributions of "computational framework demonstrating the role of associations in structured reasoning" is a bit oversold. A way to strengthen the contribution is to demonstrate the potential usage of the discovered hypothesis/pattern. Can the finding on distributional reasoning help us design better reasoning algorithm (e.g., discover self-inconsistency) or help us train models better at multi-hop reasoning? Even though the paper mentions some relevant aspects in Section 5, it lacks a proof-of-concept to showcase this is a promising way to go.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Have you conducted experiments to verify if the identified mechanism is the main function responsible for the final model prediction or how much does it contribute to it?

2. How diverse are the questions asked in the experiments? How did you assess that the dataset used contain sufficiently diverse queries for your findings to be robust?

3. Have you conducted experiment on n-hop reasoning tasks with $n>2$ to verify if your findings scale to more complex multi-hop reasoning tasks?

4. What are the average sizes of the semantic categories $c_1$ and $c_2$?
soundness: 3
strengths: The paper is well written and mostly easy to read and understand.

The proposed methodology is an interesting way for studying the challenging problem of LLM interpretability. The resulting findings provide intriguing insights into the reasoning of LLMs in two-hop composition tasks and can be very useful to the research community.
summary: The paper investigates the inner processes of LLMs on two-hop reasoning tasks. The authors identify the categories of the intermediate and final answers and study their activation in the LLMs. They find that these categories can respectively be found in the intermediate and final layers and that the connection between the intermediate and final activations can be mapped using a linear transformation. Moreover, they find that LLMs consider multiple hypotheses in parallel, with the middle layers representing the potential intermediate answers and the final layers the potential final answers.
weaknesses: Although the analysis provided in the paper is quite interesting, the main issue of the paper is its limitation to a single dataset and two-hop questions, which restricts the possibility to extrapolate the findings to larger settings. More specifically:

1. The activation of a word is independent from the rest of the logits, therefore this activation may not be the main reasoning mechanism happening inside the LLM but a byproduct or a minor function only marginally used in the total prediction. The paper does not clearly establish if the activation is the main reason for the final model prediction. It might be interesting to compute relative activations with respect to the other dimensions of the layer representation with (e.g. with softmax). 

2. The example provided in Figure 1 is a composition of two simple queries. The category mapping found could be an artifact coming from the task instead of a property of the model on two-hop reasoning tasks. The evidence provided in the experiments does not seem to eliminate this possibility.

3. Figure 4 does not include legends for the plots, making them hard to interpret. It would be good to include what each data point/plot corresponds to.

4. The experiments focus on two-hop reasoning. To assess the robustness of the findings, it would be interesting to see if they hold to n-hop queries with n higher than two. Queries recalling names could easily be modified into three-hop by asking to return the first letter of the name instead of the full name as done in Figure 1.

Meta-review: The paper investigates the internal multi-hop reasoning processes in large language models (LLMs). The authors propose that LLMs utilize a "distributional reasoning mechanism", whereby intermediate layers generate embeddings representing potential intermediate answers, which are then refined in subsequent layers. The study uses a synthetic dataset to demonstrate that these processes can be modeled using linear transformations between semantic spaces, suggesting the presence of parallel reasoning paths.

Strengths:
* Addresses a core problem in LLM research: understanding the nature of "reasoning" in LLMs. 
* A novel methodological approach to tackle this problem: an interpretable analysis of internal multi-hop reasoning processes.
* A solid, clear, well-motivated experimental design.

Weaknesses:
* The paper relies on a single synthetic dataset and only two-hop questions, which raises questions about the generalizability of the findings to more complex or natural datasets. In its current form, the findings are overly narrow.
* The paper might suffer from confirmation bias / non-falsifiable hypothesis testing. The experimental results are consistent with but stop short of proving the main hypothesis (distributional reasoning). As noted by reviewer nbdn, corroborating evidence could be provided by experiments that rule out other alternative hypotheses or that show that certain interventions motivated by the hypothesis result in changes consistent with it. But it its current form, saying that the distributional reasoning is "demonstrated" in this paper does feel overblown.

Overall, I think this is a good but borderline paper, that provides findings that are likely to be of interest to the community but whose validity and robustness is still unclear.

Decision: Reject