Title: ALPINE: Unveiling The Planning Capability of Autoregressive Learning in Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Planning

Path finding

Transformer

Autoregressive Learning

Language models
Abstract: Planning is a crucial element of both human intelligence and contemporary large language models (LLMs). In this paper, we initiate a theoretical investigation into the emergence of planning capabilities in Transformer-based LLMs via their next-word prediction mechanisms. We model planning as a network path-finding task, where the objective is to generate a valid path from a specified source node to a designated target node. Our mathematical characterization  shows that Transformer architectures can execute path-finding by embedding the adjacency and reachability matrices within their weights. Furthermore, our theoretical analysis of gradient-based learning dynamics reveals that LLMs can learn both the adjacency  and a limited form of the reachability matrices. These theoretical insights are then validated through experiments, which demonstrate that Transformer architectures indeed learn the adjacency and an incomplete reachability matrices, consistent with our theoretical predictions. When applying our methodology to the real-world planning benchmark Blocksworld, our observations remain consistent. Additionally, our analyses uncover a fundamental limitation of current Transformer architectures in path-finding: these architectures cannot identify reachability relationships through transitivity, which leads to failures in generating paths when concatenation is required. These findings provide new insights into how the internal mechanisms of autoregressive learning facilitate intelligent planning and deepen our understanding of how future LLMs might achieve more advanced and general planning-and-reasoning capabilities across diverse applications.

ALPINE: Unveiling The Planning Capability of Autoregressive Learning in Language Models

Abstract

Planning is a crucial element of both human intelligence and contemporary large language models (LLMs). In this paper, we initiate a theoretical investigation into the emergence of planning capabilities in Transformer-based LLMs via their next-word prediction mechanisms. We model planning as a network path-finding task, where the objective is to generate a valid path from a specified source node to a designated target node. Our mathematical characterization shows that Transformer architectures can execute path-finding by embedding the adjacency and reachability matrices within their weights. Furthermore, our theoretical analysis of gradient-based learning dynamics reveals that LLMs can learn both the adjacency and a limited form of the reachability matrices. These theoretical insights are then validated through experiments, which demonstrate that Transformer architectures indeed learn the adjacency and an incomplete reachability matrices, consistent with our theoretical predictions. When applying our methodology to the real-world planning benchmark Blocksworld, our observations remain consistent. Additionally, our analyses uncover a fundamental limitation of current Transformer architectures in path-finding: these architectures cannot identify reachability relationships through transitivity, which leads to failures in generating paths when concatenation is required . These findings provide new insights into how the internal mechanisms of autoregressive learning facilitate intelligent planning and deepen our understanding of how future LLMs might achieve more advanced and general planning-and-reasoning capabilities across diverse applications.

1 Introduction

Large language models (LLMs) such as ChatGPT have impressed many with their powerful capabilities across a wide range of tasks, including language processing, knowledge extraction, reasoning, planning, coding, tool use, and more [1]. However, we continue to be intrigued by the underlying mechanisms that fuel the power of LLMs. While all current LLMs are built on the Transformer architecture, which uses autoregressive learning to predict the next word in a language sequence, the overarching question remains:

Why does the process of next-word prediction give rise to intelligence?

There is no definite answer to this question yet, but researchers are approaching the problem from various angles, aiming to characterize the power and limitations of LLMs, as well as to capture their underlying acquisition, abstraction, generalization, adaptation, and reasoning mechanisms.

Recently, the mechanisms of grammar learning, knowledge manipulation, scaling laws, and arithmetic operations have been empirically uncovered [4, 3, 5, 2, 31, 11]. Furthermore, theoretical analyses have been conducted on in-context learning, chain-of-thought, and other forms of reasoning [30, 8, 7, 27]. Beyond these, LLMs' capability for planning-a fundamental component of human intelligence-has also drawn considerable attention. Planning is involved in nearly every aspect of our daily life, such as organizing a task at work, planning a trip, or seeking a mathematical proof of a theorem. Additionally, task planning plays a pivotal role in state-of-the-art LLM-empowered autonomous agents, such as HuggingGPT [18], Voyager [23], and Reflection [19]. Understanding how LLMs complete a planning task can shed light on how the seemingly low-level statistical task of next-word prediction transforms into a high-level intelligent process. Several prior studies have empirically evaluated the planning capabilities of LLMs, yielding both positive and negative results [16, 22]. However, the current results are incomplete and do not fully explain why LLMs can or cannot successfully accomplish specific planning tasks (see Appendix A for a detailed discussion of related works).

Given that planning often involves making sequential selections of next steps to achieve a desired goal, it naturally relates to the path-finding task in networks. For example, autonomous agents (e.g., HuggingGPT [18]) for scheduling API calls can be likened to finding a call path in the API call graph; a mathematical proof can be seen as a proof path from the axioms to the final theorem [21]; and a step-by-step solution to a grade-school math problem can be viewed as a path in the dependency graph among the variables [28, 29]. Many previous studies on LLM planning capabilities are related to path-finding. e.g., an LLM planning benchmark called Blocksworld [22] can be viewed as pathfinding from the initial state of the blocks to the final state in a state transition graph. Furthermore, in neuroscience, planning is often evaluated through path-finding in a maze [26]. Consequently, in this paper, we abstract planning in LLM learning as the following path-finding task: given an underlying directed graph, a Transformer architecture is provided with training data consisting of a collection of paths that specify the source node, the target node, and a path from the source to the target. The task of the language model is then to generate a path for a new source-target pair. In addition to measuring the performance of the trained model, we examine the internal weighting mechanism and the learning dynamics of the Transformer architecture during the learning and problem-solving process. This research is part of our broader project, ALPINE (Autoregressive Learning for Planning In NEtworks), which aims to answer the overarching question on the connection between the process of next-word prediction and the emergence of intelligence through the lens of planning.

Our Contributions : Our project initiates a theoretical investigation into the development of planning capabilities in Transformer-based language models by focusing on characterizing their expressiveness and learning dynamics in the path-finding task. First, in Theorem 2, we present a mathematical construction of a Transformer that encodes both the adjacency and reachability matrices of the network, thereby establishing that Transformer architectures possess the expressive capacity to complete the path-finding task. Then, in Theorem 3, we prove that when applying gradient descent to minimize the cross-entropy loss on the training data, a model based on a simplified Transformer architecture can extract the adjacency matrix and a limited form of the reachability matrix, using them to mimic human-like intelligence in path-finding. Our theoretical analysis further reveals a fundamental limitation of current Transformer architectures: they do not learn certain types of reachability, particularly transitive reachability, resulting in an incomplete ability to reason about future steps when planning. To validate our theoretical findings, we conduct extensive experiments training Transformers on the path language using autoregressive learning. First, these experiments demonstrate that Transformers achieve high accuracy in the path-finding task (Figure 3). Second, we show that it is indeed possible to extract both the adjacency and a limited form of the reachability matrices from the Transformers' weights (Figures 1,2,5,6(a)). Third, we observe a significant drop in test accuracy when the source and target nodes are connected only through concatenated path segments in the training data (Figure 6). These findings align with our theoretical analysis, confirming that current Transformers have limitations in learning transitive reachability relationships, unlike human intelligence . Finally, we validate these results on a real-world task planning benchmark, Blocksworld [22], which directly corresponds to the path-finding problem (see Appendix F).

2 Preliminaries

Throughout this paper, we use the following notations for matrices and vectors: a and A stand for a column vector and a matrix, respectively. Notations a ( i ) and A ( i,j ) denote the i th entry of vector a and the ( i, j ) th entry in matrix A , respectively. We also denote the i th row of matrix A by A ( i, :) .

2.1 Autoregressive Transformer Architecture and Loss Function

In this paper, we adopt the standard GPT architecture [17]. We use the following notation for the architecture and loss function in our analysis. Let N denote the sequence length, d the embedding size, H the number of heads, d k = d/H the embedding size per head, and M the vocabulary size. One key component of the architecture is the attention mechanism, which is formulated as:

where Q ∈ R N × d k , K ∈ R N × d k , V ∈ R N × d k are the query, key, and value matrices, respectively. Denoting X ∈ R N × d as input, the multi-head attention is computed as MHA ( X ) = Concat i ∈ [ H ] ( Attention ( XW Q i , XW i K , XW V i )) , where W Q i ∈ R d × d k , W i K ∈ R d × d k , W V i ∈ R d × d k are the learnable weight matrices for the query, key, and value matrices of the i -th head.

The feed-forward layer is a two-layer multi-layer perceptron (MLP) defined as follows:

where W 1 ∈ R d × 4 d , W 2 ∈ R 4 d × d , b 1 ∈ R 4 d , and b 2 ∈ R d are the learnable weight matrices and 1 N × x is the all-one matrix with dimension N × x . Finally, one Transformer layer is defined as:

where LN 1 and LN 2 are two layer normalizations.

With these essential components in place, we proceed to introduce the procedures of GPT. The training data consists of many sequences of tokens, where each sequence is expressed as u = ( u 1 , · · · , u N ) , in which u n denotes the token id for the n -th token in sequence u . We first represent the tokens in u by a one-hot embedding matrix U ∈ R N × M , where U ( n,u n ) = 1 and 0 elsewhere. Then there are learnable token embedding matrix W t ∈ R M × d and positional embedding matrix W p ∈ R N × d , and the input H 0 = UW t + W p ∈ R N × d . This input H 0 is fed into an L -layer Transformer, i.e., H l = Transformer ( H l -1 ) ∈ R N × d for l = 1 , · · · , L .

Finally, the output embedding goes through another layer normalization LN t , and then it is multiplied by a learnable output weight matrix W o ∈ R d × M to convert back to probability weights over all possible tokens. We calculate the output probability vector at position n , denoted by ˆ u ( n +1) , using ˆ u ( n +1) = softmax (( LN t ( H L )) ( n, :) W o ) , 1 ≤ n < N . This probability vector is used to predict the token for position n +1 , which reflects the autoregressive learning paradigm.

The adopted loss function is the cross-entropy loss for the next token prediction, given by:

2.2 Path-Planning Dataset: Syntax and Data Sources

The dataset is designed to evaluate GPT's path-planning capability on simple graphs. It is generated from a directed graph G = ( V , E ) , where V is the set of nodes, and E is the set of edges. For any two nodes u, v ∈ V , ( u, v ) ∈ E means that there is a directed edge from u to v in G . A pair of source node s and target node t is considered a valid pair if G contains at least one path from s to t .

The training dataset D contains sequences in the format ' s t s a b c t \ n', where s represents the source node token, t the target node token, s a b c t are tokens for nodes in a valid path from s to t , and \ n indicates the end of the sequence. During testing, we provide valid pairs of source and target nodes in the format ' s t '. The model is tasked with completing the remaining tokens in the sequence. The completion is considered correct if the model generates a valid path with the correct syntax.

3 Expressiveness and Learning Dynamics of Transformer Models

3.1 Expressiveness

In our path-finding task, the essential step for completing a path is to predict the next node based on the current information. It is evident that to predict the subsequent node on the path, only information related to the current node and the target node is necessary. Algorithm 1 introduces a idealized method that utilizes both the adjacency and reachability matrices of the graph. The true adjacency matrix A true and the true reachability matrix R true are defined as:

Algorithm 1 A handcrafted path-finding algorithm

1: Input: Adjacency matrix A , reachability matrix R , source node s , target node t

2: Set path P = [ s t s ] and set current node i = s

̸

3: while i = t do

4: Obtain S = { k | A ( i,k ) = 1 and R ( t,k ) = 1 }

5: Randomly sample next node k from S

6: Append k to path P , and set i = k

7: end while

8: output path P

Fact 1. Assuming that t is reachable from s , then Algorithm 1 is guaranteed to output a valid path with input A = A true and R = R true .

To illustrate the expressive capacities of the Transformer model, we first demonstrate how to manually construct a Transformer that can perform the path-finding task by simulating Algorithm 1.

Theorem 2. Given a graph G (with adjacency matrix A true and reachability matrix R true ), for every ε > 0 , there exists a 1 -layer, 1 -head, and O ( |V| ) -embedding-size Transformer model that generates a valid path for every valid source-target pair ( s, t ) with probability at least 1 -ε .

The proof involves encoding the adjacency and reachability matrices into the weights of the FFN and attention layers, respectively, while mimicking the computation of Algorithm 1 (see Appendix B).

3.2 Learning Dynamics

Having established the mathematical existence of a Transformer model capable of accomplishing path-finding in a given network, as demonstrated in Theorem 2, we now shift our focus to the following fundamental question: Can the Transformer architecture, trained on sufficient path data with an autoregressive loss as in Equation (4) and using the gradient descent (GD) method, learn the adjacency and reachability matrices and carry out path-finding similar to the idealized Algorithm 1?

Theoretically, we notice that the Transformer may not be able to learn the true adjacency and reachability matrices for the underlying graph. Instead, it can only learn the relevant information that is directly encoded in the observed training data D . Therefore, we define the observed adjacency and reachability matrices based on the training data D as follows.

Naturally, the observed adjacency matrix A obs ( D ) only records the edges ( i, k ) that appear in some path within the training data D . On the other hand, the observed reachability matrix R obs ( D ) exhibits more nuanced distinctions from the true reachability matrix. It only records that node t is reachable from node k , if the training data D contains a path (sequence) whose target node is t and k appears as a non-source node on the path. We call such pairs ( t, k ) observed reachable pairs . Noticeably, reachability through transitivity, i.e., through concatenation of path segments in D , is not observed.

Here we consider the following simplified 1-layer and 1-head Transformer structure: a) The attention weight is only on the target node (the second token), i.e., we manually set every row in softmax ( QK ⊤ √ d k ) in Eq. (1) to be a one-hot vector with the second coordinate being 1 (this is validated in our experiments shown in Figure 4), and set the positional embedding matrix W p = 0 ; b) We remove all the layer normalizations, and use FFN ( X ) = XW M instead of Eq. (2), Transformer ( X ) = FFN ( X ) + MHA ( X ) instead of Eq. (3); c) The token embedding matrix W t and the output weight matrix W o are set to be identity. The embedding size is the same as the vocabulary size ( d = M ), and we only consider the cross-entropy loss of predicting the next node.

Since there is only one layer and one head, we use W V to represent the weight of the value matrix in the attention layer. Under the above Transformer structure,

where α is the manually set attention weight matrix (every row is a one-hot vector with the second coordinate being 1 ). Therefore, the probability vector when predicting the ( n + 1) -th token is softmax ( W M ( u n , :) + W V ( u 2 , :) ) , and the prediction probability ˆ u n +1 ,k equals

Let N i,j,k be the number of times in D that: a) the current node is i ; b) the target node is j ; c) the next node is k , and let N i,j = ∑ k N i,j,k , then we have the following theorem.

Theorem 3. Under the cross-entropy loss ℓ ( D ) , for all ( i, k ) pairs, i) if ∑ j N i,j = 0 , then ∂ℓ ( D ) ∂ W M ( i,k ) is always 0; ii) if ∑ j N i,j > 0 but ∑ j N i,j,k = 0 , then ∂ℓ ( D ) ∂ W M ( i,k ) is always positive; iii) if ∑ j N i,j,k > 0 , then ∂ℓ ( D ) ∂ W M ( i,k ) is negative when W M ( i,k ) →-∞ . Similarly, for all ( j, k ) pairs, i) if ∑ i N i,j = 0 , then ∂ℓ ( D ) ∂ W V ( j,k ) is always 0; ii) if ∑ i N i,j > 0 but ∑ i N i,j,k = 0 , then ∂ℓ ( D ) ∂ W V ( j,k ) is always positive; iii) if ∑ i N i,j,k > 0 , then ∂ℓ ( D ) ∂ W V ( j,k ) is negative when W V ( j,k ) →-∞ .

Proof Sketch. By the definition of the cross-entropy loss in Eq.(4), and the prediction probability vector in Eq.(5), the total cross-entropy loss of the model (with matrices W M , W V ) is

Then we can get that: (the proof for the W V part is similar)

In case i), ∑ j N i,j = 0 implies that ∑ j N i,j,k = 0 . Hence ∂ℓ ( D ) ∂ W M ( i,k ) is always 0.

In case ii), ∑ j N i,j > 0 implies that the second term in Eq. (6) is positive, while ∑ j N i,j,k = 0 implies that the first term in Eq. (6) is 0. Hence ∂ℓ ( D ) ∂ W M ( i,k ) is always positive.

In case iii), when ∑ j N i,j > 0 and W M ( i,k ) →-∞ , the second term in Eq. (6) converges to 0, and it is smaller than ∑ j N i,j,k > 0 . Hence, ∂ℓ ( D ) ∂ W M ( i,k ) is negative when W M ( i,k ) →-∞ .

The above technical theorem directly leads to a theoretical explanation on how the model learns the adjacency and reachability information, as explained below.

Learning the adjacency matrix. Let E ( D ) denote the set of edges appearing in the training dataset D , which corresponds to the observed adjacency matrix A obs ( D ) . For any ( i, k ) ∈ E ( D ) , ∑ j N i,j,k > 0 , and for any ( i ′ , k ′ ) / ∈ E ( D ) , ∑ j N i ′ ,j,k ′ = 0 . Then from the above theorem, under the gradient descent learning paradigm, W M ( i ′ ,k ′ ) will keep decreasing (since its gradient is always positive), while W M ( i,k ) will not (since its gradient becomes negative when its value is sufficiently negative). This tends to make W M ( i,k ) higher than W M ( i ′ ,k ′ ) after training. In this way, the Transformer model learns the information about the observed adjacency matrix with matrix W M .

To facilitate comprehension, we conducted a simple experiment on the simplified Transformer, and present the results in Figure 1, In this experiment, we generate a 10-node graph, and use 3 different training datasets D 1 , D 2 , D 3 based on this graph. D 1 contains all the paths with length 1; D 2 contains

(a) True adjacency

A

true

(b)

W

M

under

D

1

(c)

W

M

under

D

2

(d)

W

M

under

D

3

0.0

0.2

0.4

0.6

8°0

1.0

2

6

8

0

2

6

H

2

6

Z

6

(a) True reachability

R

true

(b)

R

obs

(

D

1

)

(c)

R

obs

(

D

2

)

(d)

R

obs

(

D

3

)

(e)

W

V

under

D

1

(f)

W

V

under

D

2

(g)

W

V

under

D

3

.8

0.0

80

0.0

0.2

0.4

0.6

0.8

1.0

all the paths with length 1 and 20% of the paths with length higher than 1; and D 3 contains all the possible paths. Figure 1(a) is the true adjacency matrix of the graph, which is also the observed adjacency matrix for the three datasets. Figure 1(b), 1(c), 1(d) are the W M matrices with training datasets D 1 , D 2 , D 3 , respectively. Upon observation, it becomes evident that these W M matrices all successfully capture the structural information from the adjacency matrix.

Learning the reachability matrix. Similar to the process of learning the adjacency matrix, since only observed reachable pairs ( j, k ) have ∑ i N i,j,k > 0 , the gradient descent learning paradigm tends to make the W V ( j,k ) terms corresponding to observed reachable pairs ( j, k ) higher than the W V ( j ′ ,k ′ ) terms corresponding to non-observed reachable pairs ( j ′ , k ′ ) (which is either not reachable or not observed) after the training. In this way, the Transformer model captures the structural information of observed reachability matrix with weight matrix W V .

Figure 2 shows the correlation between W V and the observed reachabilities under different dataset D 's in the above experiment. Figure 2(a) is the real reachability matrix of the graph; Figure 2(b), 2(c), 2(d) are the observed reachability matrices in datasets D 1 , D 2 , D 3 , respectively; and Figure 2(e), 2(f), 2(g) are the W V matrices with training datasets D 1 , D 2 , D 3 , respectively. These illustrations show that all the weight matrices W V can satisfactorily learn the information of the observed reachabilities present in the training datasets, but cannot deduce any non-observed reachabilities.

Predicting the next node on a path. From Eq.(5), the probability vector for predicting the next node is softmax ( W M ( u n , :) + W V ( u 2 , :) ) , where u n represents the current node, and u 2 represents the target node. This resembles the procedure in Algorithm 1: it predicts the next node k such that both W M ( u n ,k ) is high (corresponding to A ( u n ,k ) = 1 ) and W V ( u 2 ,k ) is high (corresponding to R ( u 2 ,k ) = 1 ).

(a) 100 Nodes

(b) 200 Nodes

(c) 300 Nodes

(d) 400 Nodes

Generation Accuracy

Generation Accuracy

Generation Accuracy

Generation Accuracy

1 head

2 head

pea E

4 head

5 head

6 head

1 head

2 head

3 head

4 head

5 head

6 head

1 head

2 head

3 head

4 head

5 head

6 head

1 head

2 head

peau

4 head

5 head

6 head

Jafe) [

0.9550

0.9593

0.9657

0.9709

0.9638

0.9620

Jafej

20960

0.9596

0.9615

0.9602

0.9624

0.9640

Jafe) T

0.9171

0.9136

0.9121

0.9150

0.9207

0.9138

0.8736

0.8773

0.8836

0.8768

0.8806

0.8711

1.0

JaAe}

0.9730

0.9701

0.9648

0.9642

0.9674

0.9766

Jafe)

0.9675

0.9674

0.9686

0.9695

0.9664

0.9688

0.9322

00060

0.9338

0.9311

0.9339

0.9310

Jaei

0.8883

0.8843

0.8984

0.8923

0.8932

0.8925

- 0.8

Jafe} E

0.9655

0.9675

0.9651

0.9708

0.9698

0.9683

Jafe} E

0.9624

0.9666

0.9652

0.9663

0.9641

0.9679

Jafe)

0.9389

0.9414

0.9348

0.9396

0.9347

0.9409

Jafe}

0.8964

0.9002

0.8963

0.8966

0.8970

0.8923

- 0.6

Jafe]

06960

0.9637

0.9698

0.9676

0.9726

0.9727

Jafe|

0.9667

0.9624

0.9633

0.9604

0.9603

0.9640

Jafel

0.9403

0.9427

0.9386

0.9387

0.9462

0.9404

Jafe|

0.8988

0.9042

0.9063

0.9112

0.9048

0.8983

- 0.4

Jafel S

0.9642

0.9681

0.9691

0.9695

0.9681

0.9693

Jafel S

0.9637

0.9600

0.9642

0.9612

0.9640

0.9588

Jafe) S

0.9431

0.9451

0.9437

0.9416

0.9440

0.9393

Jafel S

0.8974

0.9081

0.9097

0.9032

0.9057

89060

0.2

JaAej 9

0.9649

0.9653

0.9639

0.9651

0.9639

0.9728

Jafel 9

0.9581

0.9606

0.9586

0.9610

0.9596

0.9595

Jafe) 9

0.9423

0.9470

0.9485

0.9421

0.9454

0.9479

safe) 9

0.8978

0.8987

0.9096

0.9132

0.9109

0.9071

0.0

In summary, our theoretical analysis demonstrates that a simplified one-layer, one-head autoregressive Transformer (with perfect attention) can effectively learn crucial adjacency and reachability information from the training data through gradient descent training. Moreover, it can utilize this learned information to predict the next node akin to the decision-making process of a human algorithm designer in similar scenarios. This suggests that, when confronted with the path-finding or more general planning task with a given goal, the Transformer learns the structural information to associate the next step with both the current step and the goal, enabling it to generate the subsequent task step. Nevertheless, the Transformer's limitation in learning only the observed reachability matrix-without deducing the complete reachability matrix-hints at potential constraints on the goal-oriented information it can acquire. This limitation may result in the Transformer failing to grasp novel reachability relationships derived from the transitivity of reachability relations, unlike human intelligence.

4 Empirical Evaluations: Peeking into a Trained Transformer

In this section, we conduct extensive experiments on the path-finding task using the general Transformer architecture as described in Section 2.1. The datasets are generated as described below.

The DAG is generated randomly based on two parameters: the number of nodes n , and the probability of edge p = 0 . 1 : For any 1 ≤ i < j ≤ n , there is an edge ( i, j ) ∈ E with probability p . Given the DAG, we first find all the possible reachable pairs ( s, t ) . Then these reachable pairs are separated into the training set (w.p. 0.5) and the test set (w.p. 0.5), but if edge ( s, t ) ∈ E , we always put ( s, t ) in the training set. For a reachable pair ( s, t ) in the training set, we generate m = 20 random paths that start at s and end at t , and put these m paths into the training dataset. When generating the random path, at each current node i , we find all the possible k ∈ V such that A true ( i,k ) = 1 and R true ( t,k ) = 1 (i.e., there is an edge ( i, k ) ∈ E , and k could also reach the target node t ), and uniformly choose a random one in them. Moreover, if ( s, t ) ∈ E , we always put the one-edge path ' s t s t \ n' in the training dataset to guarantee that all edges appear at least once in the training data.

4.1 Accuracy on Test Dataset

We train Transformer models on the aforementioned training dataset and subsequently evaluate the performance of these models using the pairs in the test dataset. The correctness of a model's output is determined based on its validity in terms of syntax and whether it corresponds to a valid path from s to t . In our experiments, we employ Transformer models with an embedding size of d = 120 . We conduct tests using various configurations, ranging from 1-layer and 1-head to 6-layer and 6-head, while considering different graph sizes, with number of nodes n ranging from 100 to 500. The accuracy results take average over 10000 trials, and are presented in Figure 3 (due to space limits, some results are deferred to Appendix E, and all of them are consistent with our conclusions). From these results, we make the following observations: a) When comparing the figures, the accuracy tends to decrease as the number of nodes increases; b) When examining each row, the accuracy remains relatively stable even as the number of attention heads increases; c) When examining each column, the accuracy shows at most a slight improvement as the number of layers increases.

The above observations suggest that the embedding size is the most important hyperparameter that affects the accuracy of the model. On the one hand, when the embedding size is sufficiently large compared to the graph size, even 1-layer and 1-head models perform well. This coincides with our theoretical analysis, which shows that when the embedding size equals to the graph size, the 1-layer and 1-head structure is enough to predict the next nodes accurately. On the other hand, when the

(a) 100 Nodes

(b) 200 Nodes

(c) 300 Nodes

(d) 400 Nodes

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

2

4

6

8

0

Z

4

6

8

2

4

6

8

2

4

6

8

0

(a) 100 Nodes

(b) 200 Nodes

(c) 300 Nodes

(d) Average Weight Gap

7.55.02.5

0.0

2.5

5.0

7.5

10.0

-10

7.5

0'S-

-

0.0

2.5

5.0

0

5

10

15

10

15

0

10

15

ftam abesaAy

OT

10

10

2

100 Nodes

200 Nodes

15

15

15

300 Nodes

400 Nodes

500 Nodes

2000

4000

Rerations

6000

0008

10000

embedding size is small compared to the graph size, even 6-layer and 6-head Transformers cannot achieve good performance. Because of this, in the following, we concentrate on the explainability of the 1-layer and 1-head Transformer models.

4.2 Peeking into a Trained Transformer

Attention. In our analysis, we assume that the attention is fixed on the target node. Is this true for the Transformer models learned from real data? The corresponding results are shown in Figure 4. These results are obtained by looking into the attention mechanism of the 1-layer and 1-head Transformer models, and showing the average (taking on the test dataset) matrix of softmax ( QK ⊤ √ d k ) , of which the n -th row represents the attention vector for predicting the ( n +1) -th token.

Note that the second column in these figures represents the attention weights on the second token, which corresponds to the target node in our test data. We can see that, when predicting the next tokens, almost all the attention weights are concentrated on this column, especially for those models with higher accuracy (Figure 4(a) for n = 100 and Figure 4(b) for n = 200 ). This demonstrates that indeed the Transformer model learns the correct attention for the path-finding task, and our assumption on the attention weights for the theoretical analysis is reasonable.

Adjacency Matrix. In the 1-layer and 1-head Transformers, let W M ′ (shown in Figure 5) be the matrix whose i -th row is FFN ( e ⊤ i W t ) W o +( e ⊤ i W t ) W o , where e i is the one-hot column vector that represents the token for node i . Based on the Transformer computation, intuitively this matrix is one of the components in the output that contains information related to the current node. The detailed reason for choosing this matrix is explained in Appendix D.

In Figure 5(a), the W M ′ matrix and the adjacency matrix are highly aligned: all large entries in the W M ′ matrix correspond to real edges, and all real edges correspond to large entries in the W M ′ matrix. This high accuracy is because the embedding size d = 120 is higher than the number of nodes n = 100 . If the embedding size is lower than the graph size (Figures 5(b), 5(c)), we inevitably lose some accuracy when approximating the adjacency matrix by the product of matrices with rank smaller than the graph size. Even so, there is still high relevance between W M ′ and the adjacency matrix: almost all real edges correspond to large entries in the W M ′ matrix.

In Figure 5(d), we show the gap between the average weight corresponding to edges (i.e., the average of W M ′ ( i,j ) 's with i < j and ( i, j ) ∈ E ) and the average weight corresponding to non-edges (i.e.,

the average of W M ′ ( i,j ) 's with i < j and ( i, j ) / ∈ E ) during the training process. These gaps keep increasing until convergence, suggesting that weights between edges and non-edges are more easily separated as the learning process proceeds.

Reachability Matrix. In the 1-layer and 1-head Transformers, let W V ′ be the matrix whose i -th row is ( e ⊤ i W t ) W V W o + FFN ( ( e ⊤ i W t ) W V ) W o , where e i is the one-hot column vector that represents the token for node i . Intuitively, this matrix is the remaining component in the output that contains information related to the target node. The detailed reason is also explained in Appendix D.

In Figure 6(a), we show the average weights of three different sets in the graphs: 'obs' corresponds to the W V ′ ( t,k ) 's with t ≥ k and R obs ( t,k ) = 1 ; 'real \ obs' corresponds to the W V ′ ( t,k ) 's with t ≥ k , R obs ( t,k ) = 0 but R real ( t,k ) = 1 ; and 'non' corresponds to the W V ′ ( t,k ) 's with t ≥ k and R real ( t,k ) = 0 . Here we only show the results of graphs with 100 nodes and 200 nodes, since their accuracy is high enough and their attention is quite close to being concentrated on the target node. When there are more nodes, the ability to approximate the reachability matrix is not enough for us to distinguish it. From these average weights, we can see that the Transformer learns R obs quite well, as for those terms in 'real \ obs', their weights are almost the same as those in 'non'. This echoes our analysis.

To further demonstrate that R real is not learned as good as R obs , we divide the source-target node pairs ( s, t ) in the test dataset into four categories: a) degree 0: R obs ( t,s ) = 1 ; b) degree 1: ( s, t ) is not of degree 0, while s has at least one out-neighbor node u such that ( u, t ) is of degree 0 , i.e. R obs ( t,u ) = 1 ; c) degree 2: ( s, t ) is not of degree 0 and 1, while s has at least one out-neighbor node u such that ( u, t ) is of degree 1; d) degree 3 or more: the remaining ( s, t ) pairs in the test dataset. Roughly speaking, in our analysis, for ( s, t ) pairs of degree 0 or 1, we know that there is a node u such that A obs ( s,u ) = 1 and R obs ( t,u ) = 1 . Then node u will have a large weight, indicating a high accuracy. As for ( s, t ) pairs of degree 2 or more, there is no node u such that both A obs ( s,u ) = 1 and R obs ( t,u ) = 1 . In this case, the high-weight entry when predicting the next node of s is either an adjacent node of s or a recorded node that can reach t . This should reduce the accuracy.

To see this, we check the accuracy of the Transformers on the ( s, t ) pairs of the four different categories. The results are shown in Figure 6 (b)-(d). In these figures, each row of the accuracy matrix is further divided into four sub-rows corresponding to the accuracy of degree-0 pairs, degree-1 pairs, degree-2 pairs, and degree-3 or more pairs respectively (in the graph with 100 nodes, there are no test ( s, t ) pairs in the degree-3 or more category). From these results, we can see that the accuracy for degree-2 pairs and degree-3 or more pairs is much lower than the two other categories in most cases. It indicates that, even with more parameters and a more complex structure (e.g. a 6-layer and 6-head Transformer), the Transformer model has a fundamental difficulty in generating paths for high-degree source-target pairs, namely those pairs that can only be connected by concatenating several path segments in the training dataset. This result demonstrates the validity of our theoretical analysis, i.e., after training with gradient descent on cross-entropy loss, the Transformer can only learn observed reachability, and will miss those unobserved reachability deduced from the transitivity of the reachability relation.

In summary, our extensive empirical evaluation leads to the following conclusions about the Transformer model in achieving the path-finding task: (a) With large enough embedding size, the model can achieve high accuracy in general; (b) The model achieves its performance by concentrating attention on the target nodes as intended, and learning the information on adjacency and reachability matrices, just as what a human would do and as predicted by our theoretical analysis; and (c) The model may have limitations and fail to learn high-order reachability relations through transitivity, and thus fail to generate paths derived from high-order reachability.

5 Discussion and Future Work

In summary, this paper and Project ALPINE more broadly conceptualize planning as path-finding in networks, and combine theoretical analysis of the Transformer architecture and autoregressive loss with empirical validation. Our aim is to uncover the mechanisms by which intelligence may emerge from autoregressive Transformer architectures. We analytically demonstrate that Transformers possess the expressiveness required to perform path-finding tasks and that gradient descent on crossentropy loss enables them to learn necessary-but incomplete-graph information for path-finding.

(a) Average Weights

(b) 100 Nodes

(c) 200 Nodes

(d) 300 Nodes

Generation Accuracy

Generation Accuracy

Generation Accuracy

aCesa

-1.0

ST-

-2.0

0.5

1.0

0.5

00

100 Nodes,realobs

2000

100 Nodes, obs

100 Nodes, non

4000

terations

6000

200 Nodes, reallobs

200 Nodes, non

200 Nodes, obs

8000

10000

1 layer-0.9733

2 layer

5 layer

6 layer

3 layer

4 layef

0.9828

0.9849

0.9847

0.9841

0.9529

0.9845

0.9457

0.9574

0.6085

0.4799

0.5987

0.5944

0.9507

0.6465

21560

0.5249

1 head

0.9428

0.97330.98270.9847

2 head  3 head  4 head

0.94630.95000.9523

0.98890.98750.9862

598601#860860

906009060

0.98360.98090.9871

0.98590.9859

0.94790.9548

0.9493

0.9517

0.5875

0.9499

0.6024

0.63620.6269

0.94530.95280.9470

0.5941

SEES'O

0.9598

0.9364

0.6683

0.9552

0.5939

0.5536

0.5556

0.5283

0.6187

0.6461

0.9834

0.5461

0.9849

0.9460

0.9599

0.9506

0.6279

0.9549

0.5485

0.5323

5 head

8606860

0.98780.9871

81950

0.98840.9869

0.94780.9464

0.98610.9857

0.98700.9856

0.4899

7860

0.9552

0.6053

0.9545

0.6028

0560

0.5664

0.9581

6E090

0.9427

0.9578

0.9533

17860

0.9464

0.6806

6 head

0.5303

0.9522

0.5441

0.6358

0.6673

0.9513

0.5812

1 layer-0.9628

2 layer -』

5 layer -』

6 layer -』

3 layef

4 layer -

0.9729

0.9735

0.9698

0.9696

0.9680

0.9297

0.8772

0.9425

0.9338

0.8516

0.5786

0.9471

0.9031

0.8142

0.8027

0.9355

0.7614

0.9259

0.7781

0.4541

1 head

0.7314

0.5454

9ES50

0.96720.96370.9672

2 head  3 head  4 head

0.9703

0.9422

0.9673

6660

0.8285

0.7118

0.9741

0.9417

0.9215

0.9354

0.8370

0.6345

0.9731

0.9365

0.7902

0.7955

0.9240

0.4574

08140

0.9728

59550

0.7477

0.9689

0.9717

0.9692

0.9656

0.9359

0.8623

0.5969

0.9735

0.9484

0.8874

0.5795

0.9456

0.8974

0.8658

29560

0.8170

0.6557

0.9281

0.8348

0.6815

0.9285

0.7744

0.5102

0.9696

0.9676

0.9406

0.8365

0.6716

0.9723

0.9404

0.8686

0.6375

0.9719

0.9399

0.7996

0.9687

0.9340

0.8130

0.9311

0.6852

0.9244

0.4861

0.8037

0.6078

0.7681

0.7852

5 head

0.9679

0.9683

0.9410

0.8609

0.7262

0.9741

20160

0.7330

0.9698

0.9403

0.8407

0.7502

0.9674

0.9337

0.7006

0.9655

0.4866

58260

0.7643

0.5218

0.9390

0.5967

0.9232

0.9726

6 head

0.9370

0.8973

0.6521

0.9691

0.9360

0.9050

0.5961

0.9721

0.9356

0.8402

0.8152

0.9731

0.9346

0.8147

0.9690

0.9261

0.9684

0.9269

0.4626

0.6495

0.7737

0.6488

0.6899

1 layer -0.9216

2 layer -

3 layer -

4 layer

5 layer -

6 layer -

0.9369

0.9440

0.9423

24960

C8160

0.8713

0.0498

0.0576

0.0779

0.7677

0.0406

0.7844

Z55TO

0.7299

0.6940

0.8822

0.8921

0.7485

0.7369

0.9059

0.1014

0.8949

0.8949

1 head

0.0002

0.0272

2 head

0.9224

0.0812

0.8640

0.7098

0.0140

06E60

0.8855

0.6820

0.9425

0.8940

0.7953

0.9505

0.8985

0.6861

0.1702

QE560

0.6796

0.9479

0.8991

0.7508

0.1482

0.0321

0.0248

0.8554

0.0196

0.7788

1O560

0.7547

0000

3 head

0.9188

0.8580

0.6931

0.9394

0.8885

0.7423

0.9421

0.8871

0.9474

0.8945

0.2925

0.9036

0.7902

0.1032

0.9456

0.9004

0.0535

4 head

0.9292

0.0187

0.0004

0.8574

0.7395

QEE60

0.8873

0.7272

0.9472

0.8957

0.8476

0.0682

0.9444

0.8994

0.7969

0.9474

0.8973

0.7026

0.1138

0.9546

0.9026

0.7053

0.0380

0.0199

0.0329

0.0428

6/7010

5 head

0.9238

0.8593

0.6956

0.9357

0.8891

0.7394

105610

0.8927

0.8105

0.9483

0.8987

0.81113

0.9477

0.9060

0.7349

ZEGTO

0.9500

0.9054

0.7735

0.0078

6 head

0.9213

0.8930

0.8621

0.6598

0.0232

0.9377

0.0001

0.9472

0.8956

0.8411

0.0094

0.9510

1680

0.7727

0.0784

0.9522

0.9054

0.7624

0.9474

0.9000

0.6801

0.6730

- 0.8

- 0.6

- 0.4

- 0.2

1.0

0.0

Additionally, we reveal both analytically and empirically that autoregressive training of language models has inherent limitations in the path-finding task.

Practical Implications : Our findings in LLMs for path planning may have practical implications for the training, testing, and enhancement of language models. In particular, the limitations we identified in current Transformer architectures for transitive reasoning suggest several directions for enhancing LLM frameworks to achieve more advanced and general planning-and-reasoning capabilities across diverse applications. For instance, in data generation for training, creating more diversified datasets that explicitly cover more reachability relationships may help the model achieve a higher accuracy. When evaluating a language model's planning capability, it may be beneficial to test for higher-order relationships not directly encoded in the training data but requiring chaining and concatenation to assess whether the model can perform transitive planning. Furthermore, by highlighting limitations in current language models, our study motivates future research into improved Transformer architectures, including incorporating transitivity directly into the model structure.

Challenges in Reasoning about Unobserved Reachability : Technically, the challenge in learning unobserved reachability with current Transformer architectures stems from the nature of next-token prediction loss: learning unobserved reachability incurs a higher training loss. Specifically, when predicting the next token for a given current node i and target node j , the optimal distribution for minimizing training loss should align with the observed distribution in the training dataset, i.e., Pr[ next node = k | current node = i and target node = j ] = N i,j,k N i,j (as explained in Section 3.2). Learning unobserved reachabilities requires deviating from the distribution defined by the training data, which leads to a higher training loss. Consequently, with the current training loss and Transformer architecture, the model cannot learn unobserved reachabilities, such as transitive reachability. To enable the model to learn transitivity, we may need alternative training objectives, such as path accuracy, or structural improvements to the Transformer that allow it to 'deduce' unobserved reachabilities. Conceptually, the current training data and loss objective do not provide sufficient information to teach the model transitivity or other derived relationships. Therefore, enhancing transitivity and similar capabilities may require enriching the training data, modifying the objective function, or incorporating new components into the model architecture.

Future Directions : Our investigation opens several promising directions for future research: (a) Extending our study to hyper-graphs and hyper-paths, where a hyper-edge represents scenarios requiring multiple preconditions to be met simultaneously in order to carry out the next step, as often seen in task planning and mathematical proofs. (b) Addressing the limitations of Transformers in path-finding and other planning tasks by exploring richer path-finding languages, fine-tuning, or architectural improvements to LLMs. (c) Examining connections between the abstract path-finding task and concrete planning tasks (e.g., block manipulation in Blocksworld) to understand whether, and how, Transformers abstract these tasks into path-finding frameworks. (d) Investigating in-context path-finding capabilities, where training data includes different graphs with corresponding paths, to see how Transformers learn to find new paths in new graphs. (e) Exploring the integration of chain-of-thought and backtracking capabilities into Transformers for path-finding, which may offer crucial insights into enabling these features for general search and planning tasks.

In our ongoing project ALPINE, we plan to deepen our investigation into all the aforementioned fronts. We also hope that our work will inspire more researchers to study LLMs through combined theoretical and empirical analysis, with the ultimate goal of enhancing their capabilities and understanding how human-like intelligence can be achieved through statistical learning and AI mechanisms.

References

O. J. Achiam, S. Adler, and et al. GPT-4 technical report. 2023.

Z. Allen-Zhu and Y. Li. Physics of language models: Part 1, context-free grammar. arXiv preprint arXiv:2305.13673 , 2023.

Z. Allen-Zhu and Y. Li. Physics of language models: Part 3.1, knowledge storage and extraction. arXiv preprint arXiv:2309.14316 , 2023.

Z. Allen-Zhu and Y. Li. Physics of language models: Part 3.2, knowledge manipulation. arXiv preprint arXiv:2309.14402 , 2023.

Z. Allen-Zhu and Y. Li. Physics of language models: Part 3.3, knowledge capacity scaling laws. arXiv preprint arXiv:2404.05405 , 2024.

Z. Chai, T. Zhang, L. Wu, K. Han, X. Hu, X. Huang, and Y. Yang. GraphLLM: Boosting graph reasoning ability of large language model. arXiv preprint arXiv:2310.05845 , 2023.

G. Feng, B. Zhang, Y. Gu, H. Ye, D. He, and L. Wang. Towards revealing the mystery behind chain of thought: a theoretical perspective. Advances in Neural Information Processing Systems , 36, 2023.

A. Giannou, S. Rajput, J.-y. Sohn, K. Lee, J. D. Lee, and D. Papailiopoulos. Looped transformers as programmable computers. In International Conference on Machine Learning , pages 1139811442. PMLR, 2023.

J. Guo, L. Du, and H. Liu. Gpt4graph: Can large language models understand graph structured data? an empirical evaluation and benchmarking. arXiv preprint arXiv:2305.15066 , 2023.

W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning , pages 9118-9147. PMLR, 2022.

N. Lee, K. Sreenivasan, J. D. Lee, K. Lee, and D. Papailiopoulos. Teaching arithmetic to small transformers. arXiv preprint arXiv:2307.03381 , 2023.

X. Liu, Z. Peng, X. Yi, X. Xie, L. Xiang, Y . Liu, and D. Xu. ToolNet: Connecting large language models with massive tools via tool graph. arXiv preprint arXiv:2403.00839 , 2024.

Z. Liu, Z. Lai, Z. Gao, E. Cui, Z. Li, X. Zhu, L. Lu, Q. Chen, Y . Qiao, J. Dai, et al. ControlLLM: Augment language models with tools by searching on graphs. arXiv preprint arXiv:2310.17796 , 2023.

Z. Luo, X. Song, H. Huang, J. Lian, C. Zhang, J. Jiang, X. Xie, and H. Jin. Graphinstruct: Empowering large language models with graph understanding and reasoning capability. arXiv preprint arXiv:2403.04483 , 2024.

W. Merrill and A. Sabharwal. The parallelism tradeoff: Limitations of log-precision transformers. Transactions of the Association for Computational Linguistics , 11:531-545, 2023.

I. Momennejad, H. Hasanbeig, F. Vieira Frujeri, H. Sharma, N. Jojic, H. Palangi, R. Ness, and J. Larson. Evaluating cognitive maps and planning in large language models with CogEval. Advances in Neural Information Processing Systems , 36, 2023.

A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding by generative pre-training. 2018.

Y. Shen, K. Song, X. Tan, D. Li, W. Lu, and Y. Zhuang. HuggingGPT: Solving AI tasks with ChatGPT and its friends in Huggingface. Advances in Neural Information Processing Systems , 36, 2023.

N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems , 36, 2024.

J. Tang, Y. Yang, W. Wei, L. Shi, L. Su, S. Cheng, D. Yin, and C. Huang. GraphGPT: Graph instruction tuning for large language models. arXiv preprint arXiv:2310.13023 , 2023.

T. H. Trinh, Y. Wu, Q. V. Le, H. He, and T. Luong. Solving Olympiad geometry without human demonstrations. Nature , 625(7995):476-482, 2024.

K. Valmeekam, M. Marquez, S. Sreedharan, and S. Kambhampati. On the planning abilities of large language models-a critical investigation. Advances in Neural Information Processing Systems , 36, 2023.

G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291 , 2023.

H. Wang, S. Feng, T. He, Z. Tan, X. Han, and Y. Tsvetkov. Can language models solve graph problems in natural language? Advances in Neural Information Processing Systems , 36, 2023.

L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science , 18(6):1-26, 2024.

J. C. Whittington, D. McCaffary, J. J. Bakermans, and T. E. Behrens. How to build a cognitive map: insights from models of the hippocampal formation. arXiv preprint arXiv:2202.01682 , 2022.

K. Yang, J. Ackermann, Z. He, G. Feng, B. Zhang, Y. Feng, Q. Ye, D. He, and L. Wang. Do efficient transformers really save computation? arXiv preprint arXiv:2402.13934 , 2024.

T. Ye, Z. Xu, Y. Li, and Z. Allen-Zhu. Physics of language models: Part 2.1, grade-school math and the hidden reasoning process. arXiv preprint arXiv:2407.20311 , 2024.

T. Ye, Z. Xu, Y. Li, and Z. Allen-Zhu. Physics of language models: Part 2.2, how to learn from mistakes on grade-school math problems. arXiv preprint arXiv:2408.16293 , 2024.

R. Zhang, S. Frei, and P. L. Bartlett. Trained transformers learn linear models in-context. arXiv preprint arXiv:2306.09927 , 2023.

Y. Zhang, A. Backurs, S. Bubeck, R. Eldan, S. Gunasekar, and T. Wagner. Unveiling Transformers with LEGO: a synthetic reasoning task. arXiv preprint arXiv:2206.04301 , 2022.

A Related Works

A.1 LLMs for Planning

Several recent studies have empirically evaluated the planning abilities of large language models. For instance, CogEval has introduced a variety of planning tasks set in mazes and graphs, ultimately finding no evidence to suggest LLMs grasp the intricacies of the maps or the planning tasks themselves [16]. Similarly, another study explored the Blocksworld game, a planning challenge where humans typically achieve success rates above 70% , in stark contrast to GPT-3's mere 5% success rate [22]. Our paper also proposes a novel approach by formulating a class of planning problems as path-finding on graphs, applying this model to the Blocksworld game and uncovering significant insights, as detailed in Appendix F.

Despite these seemingly negative evaluations, LLMs have shown remarkable aptitude in executing real-world planning tasks, creating the field of autonomous agents [25]. Certain applications of autonomous agents feature explicit graphs. In the tool agent HuggingGPT [18], LLMs are deployed to trigger a sequence of external APIs in response to user requests. Here, APIs are conceptualized as graph nodes, with their interrelations represented as edges, and the selection process akin to identifying a path or subgraph that aligns with user demands. This scenario is an extension of the settings discussed in this paper, where the graph is text-attributed and the objective function is evaluated through textual analysis. In addition, the application of graph search techniques has been shown to enhance the performance of tool agents significantly [13, 12]. This demonstrates that our approach of abstracting planning as path-finding in graphs is reasonable. The math agent AlphaGeometry utilizes LLMs to solve geometry problems [21]. By treating lemmas as nodes and their interdependencies as edges, the process of finding a proof of a theorem is analogous to finding a path to the theorem node in the above graph formed by possible lemma nodes and their interdependency edges. However, [21] only focuses on using LLMs to generate auxiliary constructions, and the reasoning tasks are done by a non-LLM engine. This is very different from our approach. There are no explicit graphs in other agents, such as game agents [23], embodied agents [10], and code agents [19]. The core strategy in these domains is to employ verbal reinforcement learning within LLMs. However, it is noteworthy that any dynamic programming problem characterized by deterministic state transitions can be reformulated as a shortest path problem on a graph, with states and transitions represented as nodes and edges, respectively. As a result, the area of autonomous agents is also closely related to the path-finding task investigated in this paper.

A.2 LLMs for Graphs

GPT4Graph [9] and NLGraph [24] have developed extensive frameworks for assessing LLMs in the context of graph tasks. These frameworks encompass a broad spectrum of challenges, including classic graph problems (e.g., connectivity, cycle detection, and topological sorting), graph neural network (GNN) tasks (e.g., node and graph classification), and semantic graph question answering (e.g., knowledge graph inquiries). They also explore various input formats, such as adjacency lists, edge lists, GML, and GraphML, alongside innovative prompting techniques such as few-shot, role prompting, chain-of-thought, and algorithmic prompting (e.g., stating 'we are using DFS'). These studies demonstrate that LLMs possess basic graph processing capabilities, and the choice of prompts and formats significantly influences the performance. Yet, they also reveal the models' susceptibility to spurious correlations within graphs. GPT-4, for instance, only achieves around 50% accuracy on shortest path tasks, even when utilizing complex prompts. To our knowledge, our paper presents the first theoretical analysis that identifies and explains the spurious correlations learned by transformers, partially supporting some of the negative outcomes reported in these studies.

There has also been a surge in efforts aiming at bolstering LLMs' performance on graph tasks. Innovations such as GraphGPT [20] and GraphLLM [6], which incorporate an additional GNN encoder, have shown notable improvements across the aforementioned graph tasks. GraphInstruct [14] seeks to enhance LLMs' capabilities using pure LLM approaches. This involves meticulously documenting the steps of classical algorithms (e.g., BFS and DFS) and fine-tuning LLMs to learn these graph algorithms. This method of procedural supervision has extended the capacity of LLMs in graph tasks from the complexity class TC 0 to P/poly [7]. However, while this approach has yielded performance improvements in simpler tasks such as topological sorting and connectivity, it has proven less effective for more complex challenges, e.g., finding Hamiltonian Paths.

A.3 Algorithm Simulation with Transformers

Recent theoretical investigations have shed light on the capability of the Transformer to simulate algorithms, a topic that has garnered considerable interest. From the view of discrete algorithms, Transformer models are likened to parallel circuits characterized by polynomial width and constant depth, which places them within the TC 0 complexity class (note that TC 0 ⊆ NC 1 ⊆ P ). On the other hand, despite their impressive expressiveness, the Transformer is theoretically incapable of addressing a range of P-complete problems, including the testing of Context-Free Grammar Membership [15]. However, the advent of chain-of-thought prompting has enabled the Transformer to sequentially simulate algorithms, thereby equipping them to tackle P-complete problems in domains such as arithmetic and decision-making [7]. The exploration then extends to continuous algorithms, where it has been demonstrated that the Transformer can approximate functions such as matrix inversion, Stochastic Gradient Descent (SGD), and power iterations [8]. Our study specifically applies Transformer models to simulate path-finding algorithms, presenting evidence that their expressiveness is sufficient for such tasks (Theorem 2). Nevertheless, the usage of autoregressive loss and gradient descent introduces certain limitations, which have not been studied in existing works.

A.4 Mechansims of LLMs

LLMs have demonstrated capabilities that exceed the theoretically predicted lower bounds of expressiveness. To demystify this paradox, numerous studies have employed experimental methodologies akin to those used in the physical and biological sciences. Their aim is to decode the mechanisms of LLMs. The foundational strategy is to generate controlled synthetic datasets to analyze how transformers (not necessarily LLMs) complete various tasks. Standard methods for this analysis include visualizing attention patterns to examine computational properties (such as locality and time invariance) and employing linear probing on the hidden states to determine the extent of learning. Given that the training data is synthetic and the ground-truth mappings are generally known, it becomes feasible to isolate the influence of various factors (e.g., prompting strategies, chain-of-thought reasoning, and data formatting). For example, a dataset designed for learning group operations, as detailed in [31], facilitates the exploration of how pretraining, data composition, and neural architecture influence reasoning tasks within LLMs. Similarly, the generation of synthetic context-free grammar (CFG) data, as described in [2], enables training GPT-2 models, uncovering their capacity to learn dynamic programming algorithms for parsing CFGs. Synthetic datasets focusing on biographical knowledge, presented in [3, 4, 5], probe into the mechanisms of knowledge storage, retrieval, manipulation, and the implications of scaling laws. Moreover, the work in [11] introduces synthetic datasets that aim to understand how smaller LLMs tackle basic arithmetic operations, e.g., addition, and examines the effects of few-shot prompting, pretraining, and model scaling [11]. Our work builds upon these investigations by conducting controlled experiments with a path-finding dataset, thereby shedding light on the complexities and challenges of planning in language models.

B Proof of Theorem 2

Theorem 2. Given a graph G (with adjacency matrix A true and reachability matrix R true ), for every ε > 0 , there exists a 1 -layer, 1 -head, and O ( |V| ) -embedding-size Transformer model that generates a valid path for every valid source-target pair ( s, t ) with probability at least 1 -ε .

Proof. For simplicity, we omit all layer normalizations in this construction. Suppose the input token sequence is ' s t s u 1 u 2 . . . u k ' with k ≥ 0 , where s ( = u 0 ) and t are the tokens of the source and target nodes, respectively, and nodes s, u 1 , · · · , u k form a path that can reach node t in graph G . Our objective is to construct a 1 -layer and 1 -head Transformer model that generates an out-neighbor u k +1 of u k such that there exists at least one path from u k +1 to t in G .

In essence, we utilize the attention layer to attend the output solely to the target node t . This approach allows the distribution of next token u k +1 to become a function of both the current node u k and the target node t (as formulated in Section 2). Then, by integrating the adjacency matrix A true into the FFN layer and the reachability matrix R true into the matrix W V in the attention layer, we extract row vectors R true ( t, :) and A true ( u k , :) from R true and A true , respectively, corresponding to the target node t and current node u k . By selecting proper coefficients, we can let the output be the sum of R true ( t, :) and

A true ( u k , :) . Following the softmax layer, the non-negligible entries in the final vector correspond to the feasible next nodes. With this encoding, the Transformer serves as a simulator of Algorithm 1 with input A = A true and R = R true .

Following our notation in Section 2.1, we adopt d = |V| + 2 , M = |V| + 1 and N = k + 3 . In the Transformer, there are M = |V| + 1 tokens representing the |V| nodes and the end-of-line ' \ n'. Hence, the input tokens can be represented by the one-hot embedding matrix U ∈ R N × M . We let W t = ( I M × M | 0 M × 1 ) ∈ R M × d and W p = ( 0 ( k +3) × ( |V| +1) | c 0 · e 2 ) ∈ R N × d , here e 2 represents the second unit column vector of dimension k + 3 , ( A | B ) is the notation for matrix concatenation by column, and c 0 is a positive parameter to be decided. According to the definition of the Transformer, we now have a matrix H 0 such that the first |V| +1 columns are the tokens of nodes in the sequence and the last column indicates the positions of the target node t . More specifically, we have

here e u represents the one-hot token vector for node u (with dimension M = |V| +1 ).

Then we construct the attention layer of the Transformer. We only have one head and let W K = ( 0 ( |V| +2) × ( |V| +1) | 1 ( |V| +2) × 1 ) ⊤ ∈ R d × d and W Q = √ d · I d × d . Then we can compute H 0 W K = ( 0 ( |V| +2) × 1 | c 0 · 1 ( |V| +2) × 1 | 0 ( |V| +2) × ( k +1) ) ⊤ , i.e., second rows are all c 0 's and other rows are all 0 's, and H 0 W Q = √ d · H 0 .

Therefore,

And we can compute the first part of the attention layer as

By setting c 0 → + ∞ , we obtain:

Furthermore, we set W V = ( c 1 · R true 0 |V|× 2 0 2 ×|V| 0 2 × 2 ) , where c 1 > 0 is also a parameter to be decided later. Then after the attention layer, we have a matrix as

Now we construct the feed-forward layer, which is a two-layer MLP.

For the first layer, the weight matrix W 1 is set to be,

and the bias b 1 = -c 1 · 1 4 d × 1 , which implies that 1 N × 1 b ⊤ 1 = -c 1 · 1 N × 4 d . When c 0 is large enough, the ( k + 3) th row of the matrix max ( 0 , ( MHA ( H 0 ) + H 0 ) W 1 + 1 N × 1 b ⊤ 1 )

is max ( 0 , c 1 · ( R ( t, :) | 0 1 × (3 |V| +8) ) + ( e ⊤ u k | 0 1 × (3 |V| +7) ) -c 1 · 1 1 × 4( |V| +2) ) . Since u k can reach t , in c 1 · ( R ( t, :) | 0 1 × (3 |V| +8) ) + ( e ⊤ u k | 0 1 × (3 |V| +7) ) , only the entry for node u k is c 1 + 1 while all other entries are 0 or c 1 . Therefore, the ( k + 3) th row of the matrix max ( 0 , ( MHA ( H 0 ) + H 0 ) W 1 + 1 N × 1 b ⊤ 1 ) can be arbitrarily close to ( e ⊤ u k | 0 1 × (3 |V| +7) ) . Here e u represents the one-hot token vector for node u (with dimension M = |V| +1 ).

For the second layer, we set

where c 2 is a positive parameter to be decided, and b 2 = 0 . By this way, we have

Therefore,

where e u represents the one-hot token vector for node u (with dimension M = |V| +1 ).

Then we fix c 1 = c 2 and let them be large enough. In this case, the dominant entries in ( H 1 ) ( k +3 , :) represent the nodes that are both the out-neighbor of u j and reachable to t , since those entries will have the value of 2 c 1 while other entries are at most c 1 +1 . This means that ( H 1 ) ( k +3 , :) can correctly indicates the next node u k +1 . Specifically, let W o = ( I ( |V| +1) × ( |V| +1) | 0 ( |V| +1) × 1 ) ⊤ ∈ R d × M . Then the final output approaches the following vector

where C is the number of nodes that are both the out-neighbor of u k and reachable to t . Thus, this encoding guarantees that for any ε > 0 and Q > 0 , we can always construct a 1 -layer, 1 -head, and ( |V| +2) -embedding-size Transformer that provides the correct next token with probability at least 1 -ε 2 Q by selecting large enough parameters c 0 , c 1 , c 2 .

Then we prove that there exists a Q such that this Transformer can output a correct path for every valid source and target node pair with probability at least 1 -ε . Suppose we can output all the nodes that are both the out-neighbor of the current node and reachable to the target node with the same probability in each round without any error. Then, whatever the current node is, there is a probability of at least 1 |V| |V| that the target node is reached within the next |V| generated nodes. Therefore, the target node is reached in c 3 · |V| steps with probability at least 1 -( 1 -1 |V| |V| ) c 3 , where c 3 ∈ N is a positive integer. We let c 3 = log 1 -1 |V| |V| ϵ 2 and Q = c 3 · |V| . Then, according to the Union bound, the Transformer can output a correct path in Q steps with an error rate of at most ε 2 Q · Q + ε 2 = ε .

Finally, there are two different rules (other than output a correct next node): i) when the input sequence is only ' s t ', the prediction of the next token should be the source node s ; ii) when the input sequence is ' s t s a b c t ', the prediction of the next token should be \ n. Case i) can be solved using the Transformer architecture utilizing the position information and attention to the first position; and case ii) can be solved by using the Transformer architecture utilizing the position information and attention to the second position. To maintain focus on the main construction corresponding to Algorithm 1, we omit the detailed construction for these two boundary cases.

C Proof of Theorem 3

Theorem 3. Under the cross-entropy loss ℓ ( D ) , for all ( i, k ) pairs, i) if ∑ j N i,j = 0 , then ∂ℓ ( D ) ∂ W M ( i,k ) is always 0; ii) if ∑ j N i,j > 0 but ∑ j N i,j,k = 0 , then ∂ℓ ( D ) ∂ W M ( i,k ) is always positive; iii) if ∑ j N i,j,k > 0 , then ∂ℓ ( D ) ∂ W M ( i,k ) is negative when W M ( i,k ) →-∞ . Similarly, for all ( j, k ) pairs, i) if ∑ i N i,j = 0 , then

Proof. We only prove the first part of this theorem, since the proof of the second part is almost identical.

By the definition of the cross-entropy loss in Eq.(4), and the prediction weight vector in Eq.(5) for our simplified model, the total cross-entropy loss of the model (with matrices W M , W V ) is

Then we have that

In case i), ∑ j N i,j = 0 implies that ∑ j N i,j,k = 0 . Hence ∂ℓ ( D ) ∂ W M ( i,k ) is always 0.

In case ii), ∑ j N i,j > 0 implies that the second term in Eq. (7) is positive, while ∑ j N i,j,k = 0 implies that the first term in Eq. (7) is 0. Hence ∂ℓ ( D ) ∂ W M ( i,k ) is always positive.

In case iii), when ∑ j N i,j > 0 and W M ( i,k ) converges to -∞ , then the second term in Eq. (7) converges to 0, and it is smaller than ∑ j N i,j,k > 0 . Hence, ∂ℓ ( D ) ∂ W M ( i,k ) is negative when W M ( i,k ) converges to -∞ .

D The Reason of Choosing W M ′ and W V ′

Note that in the Transformer layer, the output can be written as 3

Also noting that we have verified that the attention is concentrated at the second token, then we let X 2 = U (2 , :) W t , representing the token embedding of the target node, and X n = U ( n, :) W t , representing the token embedding of the current node.

(a) 100 Nodes

(b) 200 Nodes

(c) 300 Nodes

(d) 400 Nodes

(e) 500 Nodes

Generation Accuracy

Generation Accuracy

Generation Accuracy

1 head

2 head

3 head

4 head

5 head

6 head

1 head

2 head

3 head

4 head

5 head

6 head

1 head

2 head

3 head

4 head

5 head

6 head

Jae1

1.0

safe) [

1.0

safe) [

1.0

0.9550

0.9593

0.9657

0.9709

0.9638

0.9620

0.9602

0.9596

0.9615

0.9602

0.9624

0.9640

0.9171

0.9136

0.9121

0.9150

0.9207

0.9138

0.9730

0.9701

0.9648

0.9642

0.9674

0.9766

- 0.8

Jae} 乙

0.9675

0.9674

0.9686

0.9695

0.9664

0.9688

- 0.8

jae|

0.9322

0.9300

0.9338

0.9311

0.9339

0.9310

-0.8

Jae

0.9655

0.9675

0.9651

0.9708

0.9698

0.9683

- 0.6

Jafe}

0.9624

0.9666

0.9652

0.9663

0.9641

0.9679

- 0.6

safel

0.9389

0.9414

0.9348

0.9396

0.9347

0.9409

- 0.6

Jafe

jafe|

0.9690

0.9637

0.9698

0.9676

0.9726

0.9727

- 0.4

0.9667

0.9624

6960

0.9604

0960

0.9640

- 0.4

0.9403

0.9427

0.9386

0.9387

0.9462

0.9404

- 0.4

Jafel S

Jafe} S

0.9642

0.9681

0.9691

0.9695

0.9681

6960

0.9637

0.9600

0.9642

0.9612

0.9640

0.9588

0.9431

0.9451

0.9437

0.9416

0.9440

0.9393

- 0.2

- 0.2

- 0.2

Jafel 9

Jafe} 9

0.9649

0.9653

0.9639

0.9651

0.9639

0.9728

0.9581

0.9606

0.9586

0.9610

0.9596

0.9595

0.9423

0.9470

0.9485

0.9421

0.9454

0.9479

0.0

0.0

0.0

Generation Accuracy

Generation Accuracy

1 head

2 head

3 head

4 head

5 head

6 head

1 head

2 head

3 head

4 head

5 head

6 head

0.8736

0.8773

0.8836

0.8768

0.8806

0.8711

0T

Jafe) [

0.8063

0.8139

0.8095

0.8210

0.8165

0.8164

1.0

2 layer

- 0.8

Jafe} Z

- 0.8

0.8984

0.8923

0.8932

0.8925

0.8310

0.8486

0.8424

0.8454

0.8403

0.8510

Jafe)

0.8964

20060

0.8963

0.8966

0.8970

0.8923

- 0.6

Jafe}

0.8564

0.8508

0.8557

0.8542

0.8537

0.8610

- 0.6

4 layer

Jafe}

0.8988

0.9042

0.9063

0.9112

0.9048

0.8983

- 0.4

0.8577

0.8508

0.8609

0.8586

0.8572

0.8560

- 0.4

Jafel S

Jafe} S

0.8974

0.9081

0.9097

0.9032

0.9057

0.9068

0.8598

0.8593

0.8719

0.8631

0.8564

0.8645

0.2

- 0.2

Jafe1 9

Jafe} 9

0.8978

0.8987

0.9096

0.9132

0.9109

0.9071

0.8635

0.8676

0.8752

0.8712

0.8672

0.8679

0.0

0.0

Then we know that

It is straightforward that X n W o contains the information of the current node, and X 2 W V W o contains the information of the target node. As for FFN ( X 2 W V + X n ) W o , we choose to use its linear approximation as FFN ( X 2 W V + X n ) W o ≈ FFN ( X 2 W V ) W o + FFN ( X n ) W o . As shown in Table 1 (which takes average over all possible X 2 's and X n 's), this is a good approximation. Then we can treat FFN ( X 2 W V ) W o as the information of the target node, and FFN ( X n ) W o as the information of the current node.

Because of this, we let W M ′ be the matrix whose i -th row is FFN ( e ⊤ i W t ) W o + ( e ⊤ i W t ) W o , where e i represents the one-hot column vector for node i (with dimension M = |V| + 1 ). Note that in the simplified Transformer model of Theorem 3, there is no X n W o term, and W M ′ is the same as matrix W M . Similarly, we let W V ′ be the matrix whose i th row is ( e ⊤ i W t ) W V W o + FFN ( ( e ⊤ i W t ) W V ) W o , where e i represents the one-hot column vector for node i (with dimension M = |V| +1 ). Note that in the simplified Transformer model of Theorem 3, there is no FFN ( X 2 W V ) W o term, and W V ′ is the same as matrix W V .

E Complete Experimental Results on the Synthetic Datasets

In this section, we state the complete experimental results on the synthetic datasets. All these results are conducted on a single A100 GPU.

The accuracy results on all these tests are presented in Figure 7.

The attention results on all the 1-layer and 1-head Transformers are presented in Figure 8.

The results of W M ′ 's are shown in Figure 9.

The accuracy of the Transformers on the ( s, t ) pairs of the four different categories are shown in Figure 10.

As we can see, all these results are consistent with our conclusions.

bx

(a) 100 Nodes

(b) 200 Nodes

(c) 300 Nodes

(d) 400 Nodes

(e) 500 Nodes

(a) 100 Nodes

(b) 200 Nodes

(c) 300 Nodes

(d) 400 Nodes

(e) 500 Nodes

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

D

6

B

8

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

2

6

8

U

2

4

6

8

0

b

6

8

8

7.55.02.5

0.0

2.5

5.0

7.5

10.0

10

7.5

5.02.5

0.0

2.5

5.0

5

10

15

5

10

15

5

10

15

10

10

10

15

15

15

-10

-8

6

2.5

0.0

2.5

5.0

7.5

10.0

0

10

15

5

10

15

D

10

10

15

15

1.0

(a) 100 Nodes

(b) 200 Nodes

(c) 300 Nodes

(d) 400 Nodes

(e) 500 Nodes

Generation Accuracy

Generation Accuracy

Generation Accuracy

1 head

2 head

3 head

4 head

5 head

6 head

1 head

2 head

3 head

4 head

5 head

6 head

1 head

2 head

3 head

4 head

5 head

6 head

2 layer

3 layer

4 layer

5 layer -

6 layer

1 layer -

0.9849

0.6465

0.9529

0.6085

0.9847

0.9428

0.4799

0.9845

0.5987

87860

0.9507

0.9841

0.9733

0.9457

0.9574

0.5944

0.5249

0.9499

0.9733

0.9517

0.5875

0.9858

0.6024

0.9836

0.9463

0.6362

68060

0.5941

65860

0.9479

0.5677

0.9874

6560

SEES'O

0.9827

9560

0.5536

0.9864

0.9552

0.5939

0.9809

0.9500

0.6269

54860

0.9528

0.6683

65860

0.9548

95550

0.9598

0.9865

0.9847

0.9599

0.6461

0.9834

0.9506

0.5461

0.9871

0.9523

0.6187

79060

0.9470

0.6279

0.9849

0.9549

0.5485

0.9460

EZES'O

84860

0.9884

19860

6860

0.9842

0.9552

0.6053

0.9545

0.5618

0.9478

0.6028

0.5664

0/860

0.9581

0.6039

0.9821

0.9871

0.9869

90890

45860

95860

0.6673

0.9844

0.9513

0.5303

0.9522

0.9464

0.9578

EE560

0.5441

- 0.6

- 0.4

- 0.2

- 0.8

1.0

0.0

1 layer -

2 layer

3 layer

4 layer

5 layer

6 layer

0.8516

0.9729

0.9471

0.9031

0.9735

0.8142

86960

0.9680

0.9259

0.4541

0.9628

0.8772

0.5786

0.7314

0.9425

0.5454

0.9338

0.8027

0.5536

0.9696

0.9355

0.7614

0.7781

0.9672

0.8370

0.9731

16560

0.8285

0.7118

0.9741

0.9417

0.7780

0.9728

0.9354

0.6345

0.7902

0.5565

0.9703

0

0.7955

0.9673

0.9240

0.7477

0.4574

0.9215

0.9365

0.9637

65560

0.8623

0.9735

0.9484

0.9717

0.8974

0.8658

26960

0.8348

0.9689

0.9285

0.5969

0.8874

0.5795

0.9456

0.9362

0.8170

0.6557

0.9656

0.6815

0.7744

0.5102

9060

0.8365

0.6716

0.9723

0.6375

0.9719

0.7996

0.8037

0.9340

0.8130

0.9676

ITE60

0.7881

0.9696

0.9244

0.4861

0.9672

0.9404

0.8686

48960

0.7852

0.9679

0.9410

0.8609

0.9741

0.9390

0.7330

0.9698

0.8407

0.7502

0.9674

LEE60

0.7006

0.5967

0.9655

0.9232

0.8312

0.9683

0.9285

0.5218

0.7262

20160

0.9403

0.4866

0.7643

0.9360

0.9726

0.8973

0.9691

0.5961

0.9721

0.9356

0.8152

0.9731

0.9346

0.8147

0.6495

0.9690

0.9261

0.9684

0.4626

0.9370

0.6521

0.9050

0.8402

0.7737

0.6899

- 0.6

0.2

- 0.8

- 0.4

1.0

0.0

2 layer

3 layer

5 layer -

6 layer

1 layer -

4 layer

0.1014

0.8713

0.6940

69560

0.8822

0.7677

0.0576

0.8921

0.7485

0.0406

0.8949

0.7844

0.9522

0.8949

0.7369

0.0779

8960

0.9059

0.7299

0.9216

0.9440

0.9423

0.8640

0.7098

86560

0.8855

0.0002

0.9425

0.7953

0.0272

0.9505

0.9530

0.8977

0.6796

0.9479

0.8991

0.1482

0.9224

0.0140

0.6820

0.8940

0.8985

0.6861

0.0812

0.7508

0.8885

20000

0.8580

0.6931

0.0321

0.9394

0.7423

0.0248

0.9421

0.8871

0.8554

0.0196

0.9474

0.8945

0.7788

0.2925

10560

0.9036

0.7902

0.1032

0.9456

0.9004

0.7547

0.0004

QFE60

0.8873

0.9472

0.8957

0.8476

0.0682

0.9444

0.8994

0.7969

SESOO

0.8973

0.7026

0.9546

0.9026

0.0380

0.9292

0.8574

0.7395

0.7272

0.9474

0.1138

0.7053

0.0329

0.0428

0.9500

0.0078

0.9238

E6580

0.6956

0.0199

0.9357

0.8891

0.7394

0.9501

0.8927

0.8105

0.8113

0.0279

0.9477

0.9060

0.7349

0.1532

0.9054

0.7735

0.8411

0.0352

0.9213

0.8621

0.6598

0.9377

0.8930

0.0001

0.8956

0.0094

01560

0.8941

0.7727

0.9522

0.9054

0.7624

0.9474

0.9000

0.6801

0.0232

0.6730

0.9472

0.2116

- 0.6

- 0.8

- 0.4

- 0.2

1.0

0.0

Generation Accuracy

Generation Accuracy

1 head

2 head

3 head

4 head

5 head

6 head

1 head

2 head

3 head

4 head

5 head

6 head

1 layer

2 layer -]

3 layer

4 layer

5 layer

6 layer -

0.9098

0.9131

0.9012

0.8838

0.9004

0.2173

86160

0.8360

0.7490

0.8245

0.6193

0.8287

0.8336

0.6826

0.8356

0.6508

0.6811

0.7923

0.5528

0.4533

0.6502

0.4405

0.4537

0.4238

0.8972

0.6070

0.9049

0.8210

0.9052

0.4352

0.9121

0.6801

0.5476

18160

0.6344

0.9135

0.8331

0.6735

0.7446

0.8023

0.4321

0.6286

0.4331

0.8241

0.6848

0.8370

0.8410

0.4843

0.8850

0.8006

0.5976

0.6907

0.9049

0.8201

0.9048

0.6584

0.3651

0.9062

0.7136

0.4154

0.5923

0.9169

0.6625

0.5918

0.6569

0.5484

0.8292

0.8341

0.8505

0.8868

0.7979

0.8205

0.9061

0.9128

0.6544

0.8446

96990

0.3630

0.9138

0.8442

0.7243

0.6060

0.6679

0.8979

0.6755

0.4976

0.8255

0.6071

0.5483

0.8358

0.4535

0.6314

0.8856

0.9047

0.9111

0.8008

0.6432

0.5116

0.8234

0.6494

0.3441

0.8261

0.6921

0.4662

0.9176

0.8404

0.6842

0.4727

0.9202

0.8465

0.5971

0.3358

0.9199

0.8398

0.6876

0.3479

0.8893

0.6951

0.7986

0.6473

0.7229

0.8997

0.8154

0.6244

0.4089

0.9055

0.8389

0.6657

0.4776

0.9179

0.8390

0.7140

0.5172

0.9146

0.8399

0.5812

0.1793

0.9185

0.8432

0.6189

- 0.8

- 0.6

- 0.4

0.2

1.0

0.0

1 layer -

2 layer -

3 layer

4 layer

5 layer

6 layer

0.8615

0.8253

0.8551

0.8634

0.8761

0.7674

0.7679

96290

0.8765

ES990

0.6201

0.7107

0.4428

0.2620

0.7420

0.5641

0.2172

0.6085

0.5508

0.4947

0.7830

0.4281

0.7754

0.8414

0.8638

0.7660

0.8751

0.8799

0.7798

0.8842

0.7955

0.7198

0.4943

0.5549

0.8572

0.7566

0.5886

0.4111

0.7698

0.6413

0.5923

0.6150

0.5153

0.6451

0.6200

0.6448

0.8428

0.7203

0.4794

0.8594

0.8622

0.5832

0.8719

0.6445

0.4530

0.8738

0.6219

0.8742

0.7917

0.6198

0.7502

0.6133

0.5608

0.7693

0.6752

0.7748

0.7909

0.5409

89E90

0.6650

0.8294

0.7278

0.4777

0.5278

0.8577

0.7661

0.6270

0.4939

0.8700

0.7683

0.6499

0.6036

0.8738

0.7679

0.6881

0.6761

0.8849

0.7869

0.6472

0.5881

0.8761

0.7936

0.6426

0.6453

0.8303

0.7099

0.5226

0.8598

89E90

0.8669

0.6645

0.8731

0.7183

0.6191

0.5352

0.8793

0.7857

0.6589

0.5488

0.7533

0.4692

0.7759

0.6640

0.7765

0.7173

0.8767

0.7807

0.6994

0.8264

0.4386

0.8712

0.6380

0.2011

0.8662

0.7676

0.6138

0.8686

0.8848

0.8771

0.6197

0.7267

0.4957

0.7571

0.6460

0.7786

0.7022

0.7441

0.7900

0.6205

0.4936

0.7793

0.7107

- 0.8

- 0.6

- 0.4

- 0.2

0.0

1.0

F Path-planning in Blocksworld

To further validate the theoretical results in Section 3 and the practicability of the proposed pathfinding task, we consider Blocksworld benchmark [22]. Blocksworld is a scenario consisting of a set of blocks identified by different colors. The blocks are either placed on table or on top of another block and the task is to plan a block manipulation from the source state to the target state.

We formulate Blocksworld as a path-finding task. Here we construct a graph G BW for the case with 4 blocks, where each node represents a state of the blocks. For example, node 0 refers to the state that 'the red block is on top of the blue block, the blue block is on top of the orange block, the orange block is on top of the yellow block, and the yellow block is on the table'. G BW is a directed graph with 73 nodes, and the adjacency matrix of G BW is presented in Figure 11(a).

In the original Blocksworld task, the answer is a sequence of actions, which is equivalent to the notion of edges in G BW . We reformulated it to let the model output a path from the given source state to the given target state, only consisting of the nodes. This can be seen as a simplified version and a pure planning task. We randomly select 80% of all node pairs for training and the rest 20% for testing, and generate 50000 training sequences in the same format as introduced in Section 2. We mainly use Transformers with 1 layer and 1 head for the convenience of visualization.

F.1 Results

We first present the accuracy results during training when using different embedding sizes d ∈ { 30 , 60 , 120 } . As shown in Figure 11(d), although a smaller embedding size results in a longer time to converge, all models reach an accuracy near 100% at the end of the training.

Then, we use the same method introduced in Section 4 to visualize the attention map and the W M ′ matrix for the model with d = 120 after the entire iterations. In Figure 11(c), we can see that when predicting the tokens on the path, almost all the attention weights are on the second token which represents for the target node, demonstrating the capability of the model to learn a correct attention. For adjacency matrix, we find that the W M ′ matrix in Figure 11(b) almost perfectly aligns to the real adjacency matrix of G BW . And the weight gap (average edge weight minus average non-edge weight) for all models keeps increasing in the training process until convergence, as shown in Figure 11(e).

In addition, we present the results related to observed reachability matrix in Figure 12. Figure 12(a) shows the observed reachability in the training dataset. Although G BW is fully-connected, some reachability are not observed since we request that all training data has no cycle. Specifically, in this

(a) Adjacency Matrix of

G

BW

0

10

20

30

40

50

60

70

0

10

20

30

40

50

60

70

7.5

5.0

2.5

0.0

2.5

5.0

7.5

(b) The

W

M

′

Matrix

0

2

4

6

8

0

2

4

6

8

0.0

0.2

0.4

0.6

0.8

1.0

(c) The Average Attention

0

500

1000

1500

2000

2500

3000

Iterations

0.0

0.2

0.4

0.6

0.8

1.0

Accuravy on Test Set

d=120

d=60

d=30

0

500

1000

1500

2000

2500

3000

Iterations

0

1

2

3

4

5

6

7

Average Weight Gap

d=120

d=60

d=30

(e) Average Weight Gap

0.0

0.2

0.4

0.6

0.8

1.0

0

10

20

30

40

50

60

70

0

10

20

30

40

50

60

70

(a) Observed Reachability

0

10

20

30

40

50

60

70

0

10

20

30

40

50

60

70

5.0

2.5

0.0

2.5

5.0

7.5

10.0

(b) The

W

V

′

Matrix

0

500

1000

1500

2000

2500

3000

Iterations

1

0

1

2

3

4

Average Weight Gap

d=120

d=60

d=30

(c) Average Weight Gap

0.0

0.2

0.4

0.6

0.8

1.0

0

10

20

30

40

50

60

70

0

10

20

30

40

50

60

70

case, each of the first 24 nodes is not observed reachable to any nodes other than itself. To validate whether the Transformer has captured this information, we construct W V ′ matrix through the same method presented in Section 4. As shown in Figure 12(b), the first 24 columns of the W V ′ matrix are noticeably darker, which aligns with the observed reachability matrix in Figure 12(a). Furthermore, we plot the gap between the average weight of W V ′ on observed reachability and the average weight of W V ′ on non-observed reachability in Figure 12(c), and find that this gap keeps increasing for all models. Since there does not exist any test pairs with degree 2 or more (as defined in Section 4), we do not compare the accuracy between different degrees in Blocksworld.

In summary, our experimental results on the Blocksworld benchmark confirm our theoretical analyses (Theorem 3) and empirical results on the synthetic data, and they at least partially explain the planning capability of the Transformer on the Blocksworld scenario.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: These claims can accurately reflect the paper's contributions and scope. Details are in Section 3 and Section 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss some future research directions in Section 5. These are potential limitations of our current work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All the assumptions are provided, and the complete proofs are stated in Appendix B and Appendix C.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The code is uploaded in the supplementary material. All the information needed to reproduce the main experimental results are provided.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is uploaded in the supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Most of these details are explained in Section 2.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: It is explained that all the accuracy results take average over 10000 trials.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The computer resources is explained in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in our paper conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work mainly contributes to the theory of interpretability and explainability in Transformers. We believe the societal impacts are limited.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We believe this paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the question of whether or not a transformer-based language model can learn to plan based on next-token prediction. The authors analyse the ability of a transformer-based language model to learn in an abstracted path planning domain, and show that the model can some of the underlying structure of planning problems but has real difficulty in learning to generalise from the training data to be able to solve novel planning problems using relations such as transitivity.\n\nSTRENGTHS:\nThe overall idea of this paper is pretty strong.\n- The specific problem the authors address is important and relevant, especially determining if a transformer can learn to solve novel planning problems by inferring unobserved relations.\n- The formulation of the analysis is reasonably strong. The combination of Theorem 2 and Theorem 3 are a statement about the expressive capacity of the model, but the limitations on the learning process.\n- The experimental results do a reasonable job of supporting the theoretical analysis and claims of the paper.\n\nWEAKNESSES:\nThere are unfortunately two fairly substantial weaknesses in this paper.\n\n- Firstly, while the paper demonstrates that the specific network architecture chosen cannot deduce the existence of reachability relations that are not observed in the training data, the paper does not adequately describe why there might be any reason to think this. The result that reachability cannot be inferred without observing it is not really surprising, and the brief motivation in the third paragraph of the introduction (describing learned planning) is not really adequate. At the same time, it is a little surprising that the learner does not generalise at all to *similar* reachable concepts.\n\nThis weakness could perhaps be addressed in a revised version of the paper through a clearer introduction, and the introduction itself is a little hard to follow. I did not entirely understand where the paper was headed until I got to the end of the paper and read \"the Transformer can only learn observed reachability, and will miss those unobserved reachability deduced from the transitivity of the reachability relation.\" (To be fair, this idea is also present in the abstract, but not strongly present in the introduction.) A clearer motivation for the investigation and why it is reasonable to be unsure about what the Transformer is learning would be extremely helpful.\n\n- The second weakness is that it is not clear the extent to which the lack of inference of reachability is a problem with the specific network architecture. I was surprised that the feed forward layer is a multi-layer perceptron, rather than a graph neural network, as described by Khan et al (2020). Gama et al (2019) showed that GNNs are invariant to graph permutations which make them particularly useful in certain kinds of planning problems. The real problem is that the theoretical result is a positive result about what *is* learnable, and is supported by the experiments, but the primary conclusion of the paper is a negative result. The paper does not have a corresponding theoretical result to justify the negative result, and it is hard not to wonder if the experimental results are an accident of the specific network and training process.\n\nThe fact that the \"Transformer model has a fundamental difficulty in generating paths for high-degree source-target pairs\" may also be an accident of the network architecture, although a GNN formulation would (most likely) need to know ahead of time the degree of the node. \n\n- The experimental results are reasonable, but the specific planning domains are somewhat ad hoc and not very general. I am happy to see assessment across a range of domain sizes, but the domains are still quite limited. Future work (c) is crucial for these results to become more broadly relevant.\n\nQUESTIONS:\n- Why exactly is it not clear whether or not a Transformer can learn\n  unobserved reachability? Are there any computational structures that suggest\n  that this might be possible, or any domains with a similar kind of inference\n  process where transitivity *was* learned and used?\n\n- What possibility is there for a negative theoretical result, that a\n  Transformer of this kind could *never* learn to use transitivity and\n  therefore infer reachability?\n\n- In Figure 2, why is $R^{obs}(D_3)$ not identical to $R^{true}$, since $D_3$\n  is all possible paths?\n\nLIMITATIONS:\nThe authors did not provide a limitations section -- the technical limitations\nof the work have been addressed in the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates planning capabilities in Transformer-based language models by framing planning as a network path-finding task. It reveals that while Transformers can successfully embed adjacency and reachability matrices to perform path-finding, they struggle with transitivity in reachability, limiting their effectiveness in more complex planning scenarios. These theoretical insights are substantiated with experimental validations using both synthetic and real-world datasets, including the Blocksworld benchmark.\n\nSTRENGTHS:\n- The paper addresses a highly significant problem, which is crucial for advancing the field.\n- The experimental setup proposed is straightforward yet appears effective, which is commendable.\n- The authors have made efforts to approach the topic from both theoretical and empirical perspectives, enriching the study.\n\nWEAKNESSES:\n- The paper suffers from a lack of logical connections among the introduction, theory, and experiments. For instance, the broad question posed, \"Why does next-word prediction generate intelligence?\" lacks a clear alignment with specific aspects of their work. It is unclear which parts of their work address this question and to what extent. \n- The clarity of writing needs improvement. For example, the mention of \"Project ALPINE\" in the introduction is vague, as it does not specify what the project encompasses.\n\nQUESTIONS:\n- Could you clarify whether your project encompasses theory, algorithms, or insights? \n- Additionally, it would be beneficial to investigate whether the claims presented in your paper hold up under more realistic setup.\n\nLIMITATIONS:\n- The paper requires a rewrite to better articulate the contributions and clarify the core arguments.\n- Further investigation is necessary to assess whether the claims presented hold up under more realistic conditions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the planning capabilities of language models and provides a theoretical foundation for understanding it. The paper investigates the problem by abstracting it as a path-finding problem, showing both theoretically and empirically that transformers can embed adjacency and reachability matrices within their weights. It also highlights their limitations in handling complex planning scenarios.\n\nMain contributions: \n- This paper initiates the theoretical study of planning in autoregressive language models by abstracting it as a path-finding problem. \n- This paper shows that the transformer has the expressiveness to perform path-finding tasks, and gradient descent on cross-entropy loss cause the Transformer to learn necessary but incomplete graph information for the path-finding task. \n- This paper unveils both analytically and empirically that autoregressive training of language models has limitations in the path-finding task.\n- The paper analyzes the learning dynamics of a simplified Transformer architecture. It highlights the limitation of transformers to identify reachability relationships through transitivity. \n- The theoretical insights are supported by experiments on synthetic path-finding and a real-world planning task (Blocksworld).\n\nThe findings contribute to the broader effort of explaining the power and limitations of large language models.\n\nSTRENGTHS:\nOriginality: The approach to studying planning capabilities through path-finding in LLMs is novel and insightful. The theoretical aspects are very interesting and inspiring. \nQuality: The theoretical studies are well-supported by empirical evidence from synthetic and real-world datasets .\nClarity: The paper is well organized, with clear definitions, methodologies, and analysis of results. \nSignificance: Understand from a theoretical perspective and conduct empirical studies may help advance its capability in planning.\n\nWEAKNESSES:\n- Although the experiments are thorough. They are limited to specific datasets (synthetic and Blocksworld). The paper can benefit from broader validation across diverse planning datasets.\n- In my opinion, the practical implications of the theoretical findings are not fully explored. How to leverage the studies still requires further thinking for readers. Further elaboration could strengthen the paper.\n\nQUESTIONS:\n- How do you envision the practical applications of your findings influencing the development of future LLMs, particularly in planning? \n- Very interesting results, do you have similar studies for the other benchmark logistics in Planbench, maybe it also worth conducting this study on neutral plan benchmark. https://arxiv.org/abs/2406.04520 (note: not required for this paper, just a suggestion)\n\nLIMITATIONS:\nThe authors addressed the limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}