Title: Faith and Fate: Limits of Transformers on Compositionality
Venue: NeurIPS 2023
Accepted: yes
Keywords: Natural language processing

large language models

multi-step reasoning
Abstract: Transformer large language models (LLMs) have sparked admiration for their exceptional performance on tasks that demand intricate multi-step reasoning. Yet, these models simultaneously show failures on surprisingly trivial problems. 
This begs the question: Are these errors incidental, or do they signal more substantial limitations?
In an attempt to demystify transformer LLMs, we investigate the limits of these models across three representative compositional tasks---multi-digit multiplication, logic grid puzzles, and a classic dynamic programming problem. These tasks require breaking problems down into sub-steps and synthesizing these steps into a precise answer.  We formulate compositional tasks as computation graphs to systematically quantify the level of complexity, and break down reasoning steps into intermediate sub-procedures. 
Our empirical findings suggest that transformer LLMs solve compositional tasks by reducing multi-step compositional reasoning into linearized subgraph matching, without necessarily developing systematic problem-solving skills.  To round off our empirical study, we provide theoretical arguments on abstract multi-step reasoning problems that highlight how autoregressive generations' performance can rapidly decay with increased task complexity.

Faith and Fate: Limits of Transformers on Compositionality

Abstract

Transformer large language models (LLMs) have sparked admiration for their exceptional performance on tasks that demand intricate multi-step reasoning. Yet, these models simultaneously show failures on surprisingly trivial problems. This begs the question: Are these errors incidental, or do they signal more substantial limitations? In an attempt to demystify transformer LLMs, we investigate the limits of these models across three representative compositional tasks-multi-digit multiplication, logic grid puzzles, and a classic dynamic programming problem. These tasks require breaking problems down into sub-steps and synthesizing these steps into a precise answer. We formulate compositional tasks as computation graphs to systematically quantify the level of complexity, and break down reasoning steps into intermediate sub-procedures. Our empirical findings suggest that transformer LLMs solve compositional tasks by reducing multi-step compositional reasoning into linearized subgraph matching, without necessarily developing systematic problem-solving skills. To round off our empirical study, we provide theoretical arguments on abstract multi-step reasoning problems that highlight how autoregressive generations' performance can rapidly decay with increased task complexity.

1 Introduction

'It was the epoch of belief, it was the epoch of incredulity. ' - Charles Dickens, A Tale of Two Cities

Large-scale transformers such as ChatGPT [57] and GPT4 [58] demonstrate unprecedented capabilities [57, 74, 11, 15, 85], even noted as 'sparks of AGI' [12]. In stark contrast, the same models sometimes struggle with simple, intuitive tasks [9, 62, 40]. For instance, humans can solve 3-digit by 3-digit multiplication arithmetic after learning basic calculation rules [22, 34]. Yet, off-the-shelf ChatGPT and GPT4 achieve only 55% and 59% accuracies on this task, respectively (§3).

The striking discrepancy between the impressive successes of transformer LLMs on seemingly complex tasks and the astonishing failures on seemingly trivial tasks spark critical open questions about how to faithfully interpret their mixed capabilities. Under what conditions do transformers succeed, fail, and why? What types of errors do they make? Can transformers uncover implicit problem-solving rules or be taught to follow reasoning paths?

Seeking thorough answers to these questions remains an open research challenge. However, we offer novel insights into the fundamental limits of transformers 2 , centered around compositional

problems that require strict multi-hop reasoning to derive correct predictions. Applying step-by-step reasoning is fundamental to human intelligence [69, 68]. These compositional problems present compelling challenges for AI systems as they require combining basic reasoning operations to follow computational paths that arrive at unique correct solutions. In particular, we study three straightforward and flexible representative compositional tasks: long-form multiplication, logic grid puzzles (i.e., Einstein's puzzle [61]), and a classic dynamic programming problem.

We propose two hypotheses. First , transformers solve compositional tasks by reducing multi-step compositional reasoning into linearized path matching. This contrasts with the systematic multi-step reasoning approach that learns to apply underlying computational rules required for building correct answers [71, 37, 27]. Shortcut learning [29] via pattern-matching may yield fast correct answers when similar compositional patterns are available during training but does not allow for robust generalization to uncommon or complex examples. Second , due to error propagation, transformers may have inherent limitations on solving high-complexity compositional tasks that exhibit novel patterns. Errors in the early stages of the computational process can lead to substantial compounding errors in subsequent steps, preventing models from finding correct solutions.

To investigate our hypotheses, we formulate compositional tasks as computation graphs . These graphs break down problem-solving into submodular functional steps, enabling structured measurements of complexity and verbalization of computational steps as input sequences to language models. Moreover, we leverage information gain to predict patterns that models are likely to learn based on the underlying task distribution without the need to perform full computations within the graph.

Empirical results show that training on task-specific data leads to near-perfect performance on indomain instances and under low compositional complexity, but fails drastically on instances outside of this region. This substantial gap suggests that systematic problem-solving capabilities do not emerge from maximum likelihood training [5] on input-output sequences, even when prompted or trained with human-like reasoning steps (i.e., a linearization of computation graphs; §3.1). Models' success can be attributed, in part, to their exposure to training examples sub-graphs that involve the same computations required for solving test examples (see Section 3.2.2) In order to gain a deeper understanding of models' failures, we conduct a comprehensive analysis by decomposing their computation graphs and examining different error types. We find that while models can memorize single-step operations, they fail to compose them into correct reasoning paths, suggesting that they mostly make predictions based on shallow, rote learning rather than a deep, holistic task understanding (§3.2.3). Importantly, we provide theoretical evidence of exponential error accumulation using abstract compositional tasks. All tasks analyzed empirically in this paper are instantiations of these abstractions (§4). We argue that transformers could be inherently limited in solving compositionally complex tasks out-of-the-box 3 .

As transformers continue to make tangible real-world impacts, it is pressing to interpret their remarkable performance critically. Our work takes a realistic look at the limitations of transformers in the context of compositional tasks. To shed light on practical future steps, we identify directions for addressing these limitations, such as using transformers for tasks that could be decomposed into few reasoning steps, tasks where evaluation may afford some leniency, and using transformers in combination with planning modules or refinement methods to improve their generations. To advance language AI, fundamental innovations are required to address or complement these limitations.

2 Measuring Limitations of Transformers in Compositional Tasks

Human problem-solving skills can be conceptualized as a graph structure, where each vertex represents a partial solution and the edges represent operators that can be applied to modify these solutions. As we will outline next and illustrate in Figure 1, we use computation graphs and corresponding metrics to methodically evaluate transformers' reasoning abilities.

2.1 Computation Graph Definition

Let A be a deterministic algorithm (function), and let F A be a set of primitives (functions) the algorithm uses in its execution. Assuming the inputs x to algorithm A are given, we define A ( x ) 's static computation graph G A ( x ) . G A ( x ) = ( V, E, s, op ) is a directed acyclic graph. Nodes V

for

A

(

x

)

x

= [7,49]

multiply

1-digit

carry

concat

mod 10

sum

Color

legend:

7

4

9

28

63

2

6

8

14

1

3

4

3

343

represent all variables' values during A 's execution: each node v ∈ V has a value s ( v ) ∈ R associated. Edges E represent the function arguments involved in some computation: for each nonsource node v ∈ V , let U = { u 1 , . . . , u j } ⊂ V j be its parent nodes. Then, s ( v ) = f ( u 1 , . . . , u j ) for some f ∈ F A . Since each node v is uniquely defined by the computation of a single primitive f , we define op : V →F A as op ( v ) = f .

Let S ⊂ V be the source nodes of G A ( x ) and without loss of generality, let o ∈ V be its sole leaf node. By definition, S ≡ x and A ( x ) = s ( o ) , representing the input and output of A respectively.

To be able to train and evaluate a language model's ability to follow algorithm A wemustlinearize G A ( x ) . Since we only consider autoregressive models, this linearization must also be a topological ordering.

2.2 Quantifying Compositional Complexity using Graph Metrics

A 's representation as a computation graph G A ( x ) enables measuring task complexity from many angles.

We define a node v ∈ V 's layer number as the length of the longest path from a source node to v in the directed acyclic graph G A ( c ) . We then define the reasoning depth as the largest layer number in the graph. In computation graphs, reasoning depth is a proxy for the maximum level of multi-hop reasoning required to solve the task.

Let d S : V → N 0 be the shortest distance to any of G 's source nodes S ⊂ V . We define the reasoning width of a graph as the mode of { d ( v ) : v ∈ V } . This metric aims to measure the maximum number of variables required to maintain in parallel during the computation. Relatedly, we also define the average parallelism of a graph as the ratio between | V | and its reasoning depth. This aims to compute the average width in computation through the graph, and not just in its mode.

2.3 Predicting Surface Patterns through Relative Information Gain

When evaluating model performance, we may observe partially correct answers even in an overall incorrect response. To understand model strategies in these partial successes, we use Relative Information Gain to predict surface patterns that models are likely to recognize. We represent task T as a distribution ( X 1 , . . . , X n , Y 1 , . . . , Y m ) and measure the amount of (normalized) information gained about an output element Y j by observing a subset of input random variables X ⊂ { X 1 , . . . , X n } :

RelativeIG may be used to analyze the influence of any node in the computation graph (as defined in §2.1) with respect to a set of its ancestors; in particular, output nodes with respect to input nodes.

2.4 Exploring Three Representative Compositional Tasks: Definitions

Multiplication Multi-digit multiplication requires executing operations with numerical symbols based on procedural rules [34]. This task has multiple algorithmic solutions; in constructing computation graphs, we use the well-known O ( k 1 k 2 ) long-form multiplication algorithm for computing x · y , where x has k 1 ≤ 5 digits and y has k 2 ≤ 5 digits in base 10. See §A.1 for data construction details.

0

5

10

15

Average parallelism

0.5

0.0

0.5

1.0

1.5

GPT4 zero-shot (Multiplication)

(b)

a

GPT4zero-shot(Multiplication)

GPT4zero-shot(Puzzle)

GPT4 zero-shot(DP)

1

2

1

0.9

0.72

0.6

0.6

0.88

0.63

0.45

0.38

0.31

1.00

2

1

0.99

3

0.7

0.6

0.2

0.1

0.3

2

3

4

5

6

0.75

GPT3.5zero-shot(DP)

0.50-

3

1

0.97

0.59

4

0.8

0.2

0.1

0

0

0.51

0.21

0.12

0.08

0.04

0.25

0.00

4

0.96

0.78

0.23

0.04

5

0.1

0.1

0

0

0

2

3

4

5

6

GPT3zero-shot(DP)

5

0.91

0.54

0.09

0.01

0

6

0

0

0

0

0

0.3

0.23

0.05

0.04

0.04

1

2

3

4

5

2

3

4

5

6

2

3

4

5

6

To instantiate G A ( x ) , let F A = { one-digit multiplication, sum, mod 10, carry over, concatenation } . Source nodes S are digits of input numbers, leaf node o is the final output, and intermediate nodes v are partial results generated during execution of the long-form multiplication algorithm (see Figure 1).

Einstein's Puzzle Einstein's puzzle is a well-known logic puzzle often used as a benchmark for solving constraint satisfaction problems [61]. It involves a list of houses with different attributes (e.g., owner's name, pets), and the goal is to determine which attributes belong to each house by combining a set of pre-defined natural language clues or constraints. The solution to the puzzle is a matrix of size K × M , where K represents the number of houses and M the number of attributes. As K and M increase, synthesizing different partial solutions that satisfy individual constraints becomes highly compositionally complex. To construct the computation graph, we consider a greedy algorithm that iteratively eliminates possible solutions by filling at least one cell each time. It deterministically fills the cell(s) that requires the minimum number of clues among all current unfilled cells. We refer to this as the elimination function . See §A.2 for examples, data construction, and algorithm details.

To instantiate G A ( x ) , let F A = { elimination function } . The source nodes are the clues, all intermediate nodes are partially-filled matrices, and the output node is a fully-filled solution matrix.

Dynamic Programming Problem Dynamic programming (DP) recursively breaks down complex problems into simpler sub-problems, so problems solved using this technique are compositional. We analyze a classic relaxation of the NP-complete Maximum Weighted Independent Set problem [39]: Given a sequence of integers, find a subsequence with the highest sum, such that no two numbers in the subsequence are adjacent in the original sequence . This relaxation may be solved in O ( n ) time using DP. See the solution in §A.3. In the experiments, we restrict each integer to the [ -5 , 5] range.

To instantiate G A ( x ) , let F A = { equals, and, not , indicator function, sum, max } . Source nodes are elements of the input list, and the output node is a list that for each element indicates whether it should be selected. We select an O ( n ) algorithm since G A ( x ) 's size is proportional to A 's complexity.

3 Testing the Limits of Transformers: Empirical Evidence

Experimental Setup To understand the capabilities of LLMs, we evaluate GPT3 ( text-davinci-003 ) [11], ChatGPT ( GPT-3.5-turbo ) [57] and GPT4 ( gpt-4 ) [58] using zeroshot, few-shot, and finetuning techniques. To enable the generation of computation graphs beyond the final answers, we use the concept of scratchpads [56]. Scratchpads are a verbalization of the computation graphs (i.e., a linearized representation of a topological ordering of G A ( x ) ). Overall, we consider question-answer and question-scratchpad formats for few-shot and finetuning settings to gauge models' capabilities for learning with and without explicit reasoning. See details of additional models and experimental configurations in §B and examples of scratchpad in §A.

3.1 Testing the Limits of Transformers with Zero-shot, Few-shot and Finetuning

Limits of transformers in zero- and few-shot settings To investigate the inherent problem-solving capabilities of LLMs, we begin by analyzing models' zero-shot and few-shot performances on our compositional tasks. As shown in Figure 2, task performances deteriorate significantly from near perfection to zero with increasing complexity when measured by either problem size (Figure 2(a))or

GPT3ftw/oscratchpad

(Multiplication)

GPT3ftw/oscratchpad(DP)

1

1

1

1

0.150.10.020.02

Q

2

1

1

2

3

5

678910

3

1

0.99

0.97

GPT3ftw.SCratchpad(DP)

4

0.97

0.97

0.02

0.04

1

1

1

1

0.220.140.040.03

23

3

45678910

5

0.47

0.2

0

0.01

1

Problemsize

2

3

4

5

Problemsize

In-domain validation

OOD validation

--Early stopping

GPT3finetunedw/oScratchpad

GPT3finetuned w.Scratchpad

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0

0

50k 100k 150k 200k 250k 300k 350k 400k

Number ofsteps

0

5k

Numberofsteps

10k

15k

20k

25k

30k

average parallelism (Figure 2(b)).The trend remains the same for few-shot prompting (see §B.2). These results indicate that pre-training is in fact not sufficient to teach models how to combine basic operations to solve compositional problems, especially as problems grow more complex.

Limits of transformers with question-answer training The limited performance of models may be attributed to the lack of task-specific data during pre-training. To fully bring out models' potentials in solving these tasks, we next exhaustively finetune GPT3 with question-answer pairs. In multiplication and DP, we finetune models with all enumerations of questions up to the maximum problem size 4 within reasonable training budget, leaving out 10% for validation and 10% for testing. In puzzles, we train on a subset of all instances up to ( K,M ) ≤ (4 , 4) due to combinatorial explosion. We separately finetune GPT3 models on ∼ 1.8M multiplication pairs, ∼ 142K DP pairs, and ∼ 41K puzzle pairs (see details in §B.3). Additionally, to examine problems of different complexity, we consider different training splits based on the depth and width of computation graphs.

Figure 3 and Figure 5a show high accuracy for examples with splits seen during training, i.e., indomain . However, the performance sharply declines when evaluating unseen splits during training, i.e., out-of-domain (OOD) . Similar trends hold in all tasks (see § B.3), suggesting that systematic problem-solving capabilities do not emerge via exhaustive training on task-specific data.

Limits of transformers with explicit scratchpad training Next, we test whether we can explicitly teach models the required computational operations via scratchpads . To do so, we finetune GPT3 with question-scratchpad pairs for all tasks. We consider the same distribution splits as before. The results, presented in Figure 5b, show that once again GPT3 achieves near-perfect performance on in-distribution, but fails entirely in generalizing to OOD cases-in particular, wider or deeper computation graphs. These results indicate that even when training directly with guidance on the computation steps, models still fail to learn component operations in a generalizable manner. This observation holds for all tasks (See details in § B.4). Similarly, prompting transformers with question-scratchpad pairs enhances the performance compared to the zero-shot setting (refer to § B.5). However, this performance boost diminishes to zero as complexity increases. These findings suggest that the autoregressive characteristic of transformers, which forces them to tackle problems sequentially, presents a fundamental challenge that cannot be resolved by instructing the model to generate a step-by-step solution. Instead, models depend on a greedy process of producing the next word to make predictions without a rigorous global understanding of the task.

Limits of transformers with grokking We explore whether extended training beyond overfitting leads to improved generalization abilities, a phenomenon known as grokking [59, 53]. Due to budget constraints, we only experiment on the multiplication task. Following [53], we fine-tune GPT3 with question-answer pairs for 420K steps and separately finetune GPT3 with question-scratchpad pairs for 30K steps. Both models' training far exceeded the point at which in-domain accuracy plateaus 5 . Figure 4 shows no improvement in generalization for OOD cases beyond the overfitting point, even

Multiplicationw/oScratchpad

Multiplicationw/oScratchpad

Multiplicationw.Scratchpad

Multiplicationw.Scratchpad

1.0

GPT3 finetuned

GPT3 five-shot

1.0

GPT3 finetuned

GPT3 five-shot

1.0

GPT3 finetuned

GPT3 five-shot

1.0

GPT3 finetuned

GPT3 five-shot

0.8

GPT3 zero-shot

0.8

GPT3 zero-shot

0.8

GPT3 zero-shot

0.8

GPT3 zero-shot

0.6

2

cura

0.6

0.6

2

cura

0.6

0.4

A

0.4

0.4

AC

0.4

0.2

0.2

0.2

0.2

0.0

in-domain data

0.0

in-domain data

0.0

in-domain data

0.0

in-domain data

0

5

10

15

20

0

5

10

15

20

0

5

10

15

20

0

5

10

15

20

Graph depth

Graphwidth

Graph depth

Graphwidth

(a) Results on question-answer pairs.

(b) Results on question-scratchpad pairs.

after extensive training periods. We hypothesize that the absence of grokking may be due to the level of difficulty of the task. We speculate that increased task difficulty significantly impedes learning a well-structured representation, which, according to [47], aligns with achieving grokking. Even if grokking were to emerge through more prolonged training, such an approach would prove inefficient and unscalable. Future work is required to accurately explain when and how grokking occurs.

3.2 Breaking Down Successes and Failures of Transformers

3.2.1 Information Gain Explains Where Transformers Partially Excel

At times transformers predict partially correct answers even when the overall response is incorrect. We speculate that this may be due to particularities in the task distribution that allow for guessing partial answers without performing the full multi-step reasoning that the task requires.

Using relative information gain (defined in §2.3), we can predict surface patterns that a model is likely to learn and contrast them empirically. For multiplication, relative information gain shows that the first digit (two digits) of the output highly correlates with the first digit (two digits) of each input number (see §C.1). Hence, this spurious pattern is likely to be learned by a model. Similarly, the prediction of the last digit (or two digits) of the output is observed to solely rely on the last digit (or two digits) of each input number. This pattern holds true due to the principles of modulo arithmetic, which ensures the validity of this relationship in all cases. Empirically, we verify that models indeed learn the patterns we predicted and other patterns as well (e.g., order of magnitude of the answer, number of trailing zeros for multiplication) in all the settings with and without scratchpad. See details for multiplication, plus dynamic programming task analysis in §C.

These experiments suggest that if an output element heavily relies on a single or a small set of input features, transformers are likely to recognize such correlation during training and directly map these input features to predict the output element in testing, without going through the rigorous multi-hop reasoning and giving a false illusion of performing compositional reasoning.

3.2.2 Transformers Reduce Multi-Step Compositional Reasoning into Linearized Subgraph Matching

We now explore whether models' correct predictions on unseen test data are due to learning the underlying algorithm or, instead, explainable by exposure to similar training examples. We hypothesize that, beyond simple memorization, transformers largely rely on pattern matching for solving these tasks. To test this, we calculate the average frequency with which partial computations needed to solve an instance appear in the training data, for both correctly and wrongly predicted examples.

Given a model-generated computation graph ̂ G A ( x ) we analyze how often the full computation of each node v ∈ ̂ V is seen in training. We define v 's full computation as the subgraph induced by all ancestors of v including v , denoted FC ̂ G A ( x ) ( v ) . We say that FC ̂ G A ( x ) ( v ) is seen during training if FC ̂ G A ( x ) ( v ) ∼ = FC G A ( x ′ ) ( w ) for some computation graph G A ( x ′ ) in training, and for some w ∈ V . We characterize complexity of a full computation subgraph by its depth, as defined in §2.1.

Figure 6 shows that full computation subgraphs appear significantly more frequently in the training data for correctly predicted test examples than for incorrectly predicted ones, for both the multi-

training data

Fine-tuned GPT3 - Multiplication

training data (log)

Fine-tuned GPT3 - Dynamic Prog.

Correct Final Answer

Incorrect Final Answer

Fine-tuned GPT3 - DP

10M

5M

0

6

8

10

0.1

0.01

full computation subgraph depth

5

10

15

20

Fully Correct

Local Error

Propagation Error

Restoration Error

Five-shot GPT4 - DP

Fine-tuned GPT3 - DP

Five-shot GPT4 - Multiplication

Five-shot GPT4 - Puzzle

frequency

0.8

10000

0.2

0

10

15

10

20

30

0

10

15

20

Fine-tuned GPT3-Multiplication

Fine-tuned GPT3 -Puzzle

frequency

0.8

0.6

0.4

0.2

5

10

10

15

25

0

5

10

15

graph layer number

graph layer number

graph layer number

plication and DP task (both frequencies tend to zero for large depths since we ensured a disjoint train/test split). This high correlation suggests that pattern matching-and not general reasoning capabilities-may be the cause behind correct model outputs. This type of learning could be largely effective when the compositional complexity of tasks is low but it becomes less efficient when tasks are increasingly complex. This may elucidate the observed performance gain in low-complexity and in-domain cases and the striking performance drop in OOD and highly complex cases.

3.2.3 What Types of Errors do Transformers Make at Different Reasoning Depths?

For clearer understanding of where transformers fall short, we analyze the types of errors that transformers make for nodes at different layers in the computation graph. For every input x , we compare the ground truth computation graph G A ( x ) with the (possibly incorrect) model-generated computation graph ̂ G A ( x ) . Weconsider a node v as having a correct value if and only if s ( v ) = ̂ s ( v ) . 6 . We consider a node v to be derived from a correct computation if given that U = { u 1 , . . . , u k } are the immediate predecessors of v in ̂ G A ( x ) and that ̂ op ( v ) = f , we have that f ( u 1 , . . . , u k ) = ̂ s ( v ) . Note that the notion of correct computation is independent of G , and that a node v derived from a correct computation may not have the correct value if an error occurred in some of its ancestors.

We classify each node v ∈ ̂ V into one of four categories. Node v is fully correct if v and its ancestors have correct values and are derived from correct computations. If a node v is not fully correct, its error can be of the following types: v has a local error if its parent nodes have correct values but v is derived from an incorrect computation (i.e., a one-hop reasoning error); v has a propagation error if v is derived from a correct computation but some of its parent nodes have incorrect values; v has a restoration error if it has a correct value but is derived from an incorrect computation.

Figure 7 shows results for few-shot GPT4 and fine-tuned GPT3 with scratchpad, with respect to graph layer number for each node. In all settings, the ratio of fully correct nodes is almost perfect but sharply decreases toward zero with increasing graph layers. Moreover, the ratio of propagation errors is usually higher than the ratio of local errors. Both phenomena suggest that models are able to correctly perform single-step reasoning, potentially due to memorizing such single-step operations during training, but fail to plan and compose several of these steps for an overall correct reasoning.

Both the DP and the puzzle tasks have a high ratio of restoration errors, suggesting memorization since correct outputs are produced despite incorrect computations. There are signs of memorization even when restoration errors are near zero: 82.3% of the final correct answers for 4-digit by 2-digit multiplications (a setting unseen during training) had at least one error in the computation graph, but still produced correct answers. These patterns are possibly due to high frequency of (input, output) multiplication pairs in the pretraining data, in contrast to intermediate reasoning steps.

4 Error Propagations: The Theoretical Limits

Experiments (§3) highlight the limitations of current transformers in handling complex, multi-step reasoning tasks. Concretely, we show that errors rapidly escalate as the problem size grows (§3.2.3). Here, we aim to provide theoretical insights into why autoregressive transformer LLMs can perform significantly worse in compositional tasks as the problem size increases, making explicit the different ways in which compounding stochastic errors affect final performance. We argue using stylized examples that transformers may be too limited to solve compositionally complex tasks. Formal statements and full proofs are provided in §D.

Algorithms designed to solve compositional tasks typically involve multiple independent applications of a function and/or iterated applications of the same function. A transformer executing such an algorithm acts as an estimator of these functions. In this context, we examine the probability of such an estimator reaching the correct answer as the problem size increases. We first consider a scenario where a transformer estimates an algorithm requiring n independent applications of a function:

̸

̸

Proposition 4.1 (informal) . Let f n involve the combination h n of n independent applications of a function g . Let ̂ f , ̂ g , ̂ h n be their estimators. Assume that ̂ h n is a perfect estimator of h n and that h n has low collision, with c n being an upper bound of h n 's collision rate ( c n < c ∀ n , with c ≪ 1 ). If P ( g = ̂ g )= ϵ > 0 where ̂ g 's errors are independent, then P ( f n = ̂ f n ) > 1 -c n -(1 -ϵ ) n · (1 -c n ) . This implies that P ( f n = ̂ f n ) decreases exponentially as n increases, with lim inf n → + ∞ P ( f n = ̂ f n ) ≥ 1 -c . Moreover, if c n ≤ βα n for some α ∈ (0 , 1) , β > 0 , P ( f n = ̂ f n ) tends exponentially to 1 as n increases. Prop. 4.1's proof (§D.1) shows the rate of convergence is exponential, thus concluding that transformers will rapidly fail with increasing n . Let's now analyze the iterated application function scenario.

̸

̸

The argument is as follows. Let s n := P ( f n = ̂ f n ) , where s 1 = 1 -ϵ by definition. Derive s n ≤ (1 -ϵ -c ) · s n -1 + c using law of total probability. Then, prove by induction a non-recursive upper bound for s n with limit c c + ϵ when n → + ∞ . See formal statement and derivation in §D.2.

̸

Proposition 4.2 (informal) . Let f n ( x )= g n ( x ) involve the repeated application of g . Assume that the probability of recovering from a mistake due to the randomness of applying the estimator on an incorrect input has probability at most c . If P ( g = ̂ g ) = ϵ > 0 , then P ( f n = ̂ f n ) decreases exponentially with n . Precisely, P ( f n = ̂ f n ) ≥ 1 -(1 -ϵ -c ) n -1 (1 -ϵ -c/ ( c + ϵ )) , implying lim inf n → + ∞ P ( f n = ̂ f n ) ≥ 1 -c/ ( c + ϵ ) .

̸

Moreover, repeated applications of a function often imply unbounded errors: if g ( x ) can be expressed as an affine transformation Fx + c , then it may be viewed as a first-order vector autoregression, which are known to be unstable when | λ | ≥ 1 for at least one λ eigenvalue of F [31, Prop. 10.1]. While we make these arguments with affine maps, similar behaviors, possibly even more acute, could occur with nonlinear maps [25]-but their study is beyond the scope of this paper.

Prop. 4.2's proof also shows an exponential rate of convergence. Note that if c ≪ ϵ then lim inf n → + ∞ P ( f n = ̂ f n ) ≈ 1 . It is reasonable to assume c ≪ ϵ when g has low collision, since c represents the probability of the estimator ̂ g ( y ) arriving at the correct output g ( x ) by chance when given the wrong input y = x . More details in §D.3.

̸

In Prop. 4.2's current form, we implicitly assume that there is a single valid reasoning for each input since g is a function. We can potentially generalize this assumption with a state-transition framing, where the probability of transitioning from a valid state to an invalid one is ϵ , and the probability of recovering from an invalid state is at most c . See formal statement in D.2.

All tasks evaluated in the present work can be seen as instances of the results just proven. Prop. 4.1 directly applies to multiplication, since m -byn digit multiplication can be seen as n independent instances of m -by1 digit multiplication (see Cor. D.1). Prop. 4.2 directly applies to the recursive function of the dynamic programming task, as well as to m -by1 digit multiplication, and to the puzzle through its elimination function (details in D.3). They are also all low collision settings.

Note that Prop 4.1 and 4.2 apply to any high-performant estimator of reasoning tasks. We focus on out-of-the-box transformers to align with the scope of our experiments and with the goal of framing empirical results. In §5, we discuss how these propositions may inform future research directions.

̸

̸

̸

̸

5 Discussion

Collapsed Compositionality and Robustness Implications Transformers today demonstrate undeniably powerful empirical results. Yet, our study suggests that they may have fundamental weaknesses in certain intellectual tasks that require true multi-step compositional operations such as multiplications and logic puzzles. Our careful study based on the computation graph and analyses demonstrates that transformers can often solve multi-step compositional problems by collapsing the depth of the compositional operations via analogical pattern matching. More broadly, our findings suggest that the strong performance of transformers should be taken with a certain grain of salt: Despite initially appearing challenging, certain tasks may not possess the inherent compositionality they seem to have. This is due to the fact that desired solutions could be readily derived from input-output sequences present in the training data, allowing for shortcut pattern matching to produce acceptable solutions. However, such an approach can ultimately result in poor generalization as shown in our study. For example, fine-tuning GPT3 on our tasks both with and without explicit reasoning graphs shows that models' learning fails to generalize beyond levels of complexity seen in training.

Theoretical Findings and their Empirical Implications The proofs presented in §4 show that, under reasonable assumptions, the probability of incorrect predictions converges exponentially to ≈ 1 for abstract compositional tasks. Importantly, these proofs apply to autoregressive LMs in general. Our insights indicate that the current configuration of transformers, with their reliance on a greedy process for predicting the next word, constrains their error recovery capability and impedes the development of a comprehensive global understanding of the task. Building on these findings, we suggest several empirical strategies for harnessing the potential of transformers. Firstly, transformers may be employed in ways that require chaining only a few compositional steps to reach a solution rather than lengthy reasoning steps (e.g., [35]). Secondly, transformers may be best suited for compositional tasks where evaluation metrics can afford some leniency; for example, finding approximate solutions that do not require executing the whole graph, such as identifying the most significant digit in a multiplication. Finally, we suggest augmenting transformers with planning modules as well as using refinement methods, that can iteratively improve their generations [82, 48].

Call for Broad Participation to Investigate Limitations Identification of limitations is an important step towards achieving greater robustness. Our study suggests fundamental limitations that impede transformers from fully mastering certain compositional operations. However, we acknowledge that due to our compute budget constraints as well as limited access to the largest language models such as GPT4, we are unable to push the empirical limits of transformers even further in terms of training data size and number of epochs. We invite the broader research community, particularly those with more extensive resources at their disposal, to investigate these possibilities further.

6 Related Work

Reasoning abilities in transformer LLMs Recently, transformers [11, 58, 57, 17, 16, 63, 73, 74] have demonstrated impressive reasoning abilities across a wide range of tasks, even outperforming humans in certain cases [79, 28, 15, 85]. This success has been largely attributed to the scaling effect, where larger models and training datasets result in improved performance [38, 33, 1]. However, these models have also been shown to struggle across multiple domains [32], including algorithmic reasoning [78], commonsense reasoning [62, 40], theory of mind [65], planning [76], logical reasoning [70], and ethical reasoning [36]. These difficulties have motivated us to take a step back and thoroughly examine both the successes and failures of transformers from empirical and theoretical perspectives on compositional reasoning tasks.

Challenges of transformers in compositional tasks Transformers perform fairly well in singlestep reasoning tasks [70], but face challenges when it comes to effectively combining multiple steps to solve compositionally complex problems [84, 55, 66, 81]. Recent research has focused on overcoming these limitations through various approaches. First, fine-tuning transformers to directly generate the final answer while keeping the reasoning implicit [7, 18]. Second, encouraging transformers to generate reasoning steps explicitly within a single generation [55, 80, 44, 42]. For example, Nye et al. [55] and Zhou et al. [86] used scratchpads to teach transformers how to perform algorithmic reasoning tasks such as addition by splitting the task into intermediate steps [44, 80]. Further, leveraging LLMs to generate each reasoning step iteratively via a selection and inference mechanism [20, 19, 72].

Lastly, choosing a training split that maximizes the number of observed patterns between the train and test data [10], or diversifying in-prompt examples to cover the maximum of patterns [41], ultimately enhancing generalization. The primary focus of these studies is to enhance model performance on compositional problems without striving for complete mastery. In contrast, our work explores the fundamental limits of vanilla transformers in achieving full mastery, striving for 100% performance in both in-domain and OOD settings. Our findings show that reaching full mastery is inherently challenging, providing insights into the complexities involved.

Challenges of transformers in generalization Extensive research has been done to investigate the generalization capabilities of transformers [3, 54, 26, 64, 8, 46]. This encompasses various facets of generalization, including easy-to-hard generalization [67, 4], length generalization [2, 60, 13, 54, 8], and generalization on symbolic mathematical integration [83]. Schwarzschild et al. [67] and Bansal et al. [4] employ weight-tied neural networks to generalize from easy to hard examples. Liu et al., [45] found that shallow transformers learn shortcuts during training, leading to poor OOD generalization. Razeghi et al. [64] revealed a positive correlation between the frequency of training terms and their test performance. Building upon this line of inquiry, we present a more rigorous examination of sub-graph matching between training and test instances for complex compositional tasks where we demonstrate how pattern matching can hinder generalization. We complement our empirical results with theoretical insights on transformers' limits.

Grokking The phenomena of models' gaining generalization capabilities when training significantly beyond overfitting, known as grokking was recently introduced in [59]. Subsequent works focus on characterizing when and why grokking arises: [47] show that perfect generalization in an arithmetic addition task happens when there is sufficient data to determine the appropriate structured representation, later extended to sparse parity in [52] where a sparse subnetwork of neurons is shown responsible for generalization behavior. Recently, [77] propose that grokking occurs when a task admits a generalizing and a memorizing solution, and the former is slower to learn. In this present work, our aim is not to explain grokking but rather to observe its emergence. We do not observe grokking arising in the context of multiplication, and we leave it to future work to explore whether this may be due to task difficulty hindering the learning of well-structured representations.

Transformers' theoretical expressiveness Lin et al. [43] study autoregressive models' limitations from a computational complexity theory perspective. Transformer-specific work has focused on quantifying the class of problems that (not necessarily autoregressive) transformers can express assuming perfect parameters [51, 50, 14, 49, inter alia]. All tasks analyzed in our work belong to a class expressible by transformers, suggesting that known upper bound might not be tight. Importantly, Hahn [30] shows that transformers cannot robustly model noncounter-free regular languages even when allowing infinite precision. In contrast, our focus is on error accumulation, which enables to investigate if reasoning tasks theoretically solvable by transformers are likely to be solved by them.

Additional literature and societal impact discussion can be found in §E.

7 Conclusions

On a broader scope, as transformers continue to gain widespread deployment with significant realworld impacts, it is ever more urgent to understand their successes and failures. Our study critically investigates transformers' limitations and emphasizes the need to develop models capable of robust generalization and systematic problem-solving. By examining the compositional capabilities of these models, we aspire to work towards more reliable AI systems that excel not only in tasks where abundant training examples are sufficient, but also in cases requiring precise compositional reasoning.

8 Limitations

We focus on analyzing compositional reasoning capabilities through the lens of computation graphs. Although they are a useful way to systematically represent rigorous reasoning processes, it is important to note that for the scratchpad approach, we are limited to only establishing a correlation between the model generation and its preceding context, as we cannot inspect the exact tokens model attends to when making the prediction. This limitation arises from our lack of access to the activations of the studied models. Furthermore, we posit that alternative approaches to linearizing reasoning processes may yield different performances and provide opportunities for further exploration.

References

Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for generative mixed-modal language models. CoRR , abs/2301.03728, 2023.

Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring length generalization in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 38546-38556. Curran Associates, Inc., 2022.

Cem Anil, Yuhuai Wu, Anders Johan Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Venkatesh Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring length generalization in large language models. In Advances in Neural Information Processing Systems , volume 35, pages 38546-38556. Curran Associates, Inc., 2022.

Arpit Bansal, Avi Schwarzschild, Eitan Borgnia, Zeyad Emam, Furong Huang, Micah Goldblum, and Tom Goldstein. End-to-end algorithm synthesis with recurrent networks: Extrapolation without overthinking. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 20232-20242. Curran Associates, Inc., 2022.

Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. In Advances in Neural Information Processing Systems , volume 13. MIT Press, 2000.

D. Bertsekas. Abstract Dynamic Programming: 3rd Edition . Athena scientific optimization and computation series. Athena Scientific., 2022.

Gregor Betz, Christian Voigt, and Kyle Richardson. Critical thinking for language models. In Proceedings of the 14th International Conference on Computational Semantics (IWCS) , pages 63-75, 2021.

Satwik Bhattamishra, Kabir Ahuja, and Navin Goyal. On the practical ability of recurrent neural networks to recognize hierarchical languages. In Donia Scott, Núria Bel, and Chengqing Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020 , pages 1481-1494. International Committee on Computational Linguistics, 2020.

Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yaojie Lu, and Ben He. ChatGPT is a knowledgeable but inexperienced solver: An investigation of commonsense problem in large language models. CoRR , abs/2303.16421, 2023.

Ben Bogin, Shivanshu Gupta, and Jonathan Berant. Unobserved local structures make compositional generalization hard. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 2731-2747, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems , volume 33, pages 1877-1901, 2020.

Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4. CoRR , abs/2303.12712, 2023.

Mirelle Bueno, Carlos Gemmel, Jeffrey Dalton, Roberto de Alencar Lotufo, and Rodrigo Frassetto Nogueira. Induced natural language rationales and interleaved markup tokens enable extrapolation in large language models. CoRR , abs/2208.11445, 2022.

David Chiang, Peter Cholak, and Anand Pillay. Tighter bounds on the expressivity of transformer encoders. Proceedings of the 40th International Conference on Machine Learning , 202:55445562, 2023.

Jonathan H Choi, Kristin E Hickman, Amy Monahan, and Daniel Schwarcz. ChatGPT goes to law school. Available at SSRN , 2023.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. CoRR , abs/2204.02311, 2022.

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-finetuned language models. CoRR , abs/2210.11416, 2022.

Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers as soft reasoners over language. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence , pages 3882-3890, 2021.

Antonia Creswell and Murray Shanahan. Faithful reasoning using large language models. CoRR , abs/2208.14271, 2022.

Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning. In International Conference on Learning Representations , 2023.

Leonardo De Moura and Nikolaj Bjørner. Z3: An efficient smt solver. In Tools and Algorithms for the Construction and Analysis of Systems: 14th International Conference, TACAS 2008, Held as Part of the Joint European Conferences on Theory and Practice of Software, ETAPS 2008, Budapest, Hungary, March 29-April 6, 2008. Proceedings 14 , pages 337-340. Springer, 2008.

Stanislas Dehaene, Nicolas Molko, Laurent Cohen, and Anna J Wilson. Arithmetic and the brain. Current opinion in neurobiology , 14(2):218-224, 2004.

B. Delyon and A. Juditsky. On small perturbations of stable markov operators: Unbounded case. Theory of Probability & Its Applications , 43(4):577-587, 1999.

Persi Diaconis and David Freedman. Iterated random functions. SIAM review , 41(1):45-76, 1999.

R. Douc, E. Moulines, and D. Stoffer. Nonlinear Time Series: Theory, Methods and Applications with R Examples . Chapman & Hall/CRC Texts in Statistical Science. Taylor & Francis, 2014.

Yann Dubois, Gautier Dagan, Dieuwke Hupkes, and Elia Bruni. Location attention for extrapolation to longer sequences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 403-413, 2020.

Jonathan St BT Evans. Bias in human reasoning: Causes and consequences. Lawrence Erlbaum Associates, Inc, 1989.

Hao Fu, Yao; Peng and Tushar Khot. How does gpt obtain its ability? tracing emergent abilities of language models to their sources. Yao Fu's Notion , Dec 2022.

Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence , 2(11):665-673, 2020.

Michael Hahn. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics , 8:156-171, 2020.

James Douglas Hamilton. Time series analysis . Princeton university press, 1994.

Chadi Helwe, Chloé Clavel, and Fabian M. Suchanek. Reasoning with transformer-based models: Deep learning, but shallow reasoning. In Danqi Chen, Jonathan Berant, Andrew McCallum, and Sameer Singh, editors, 3rd Conference on Automated Knowledge Base Construction, AKBC 2021, Virtual, October 4-8, 2021 , 2021.

TomHenighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for autoregressive generative modeling. CoRR , abs/2010.14701, 2020.

James Hiebert. Conceptual and procedural knowledge: The case of mathematics . Routledge, 2013.

Albert Qiaochu Jiang, Sean Welleck, Jin Peng Zhou, Timothee Lacroix, Jiacheng Liu, Wenda Li, Mateja Jamnik, Guillaume Lample, and Yuhuai Wu. Draft, sketch, and prove: Guiding formal theorem provers with informal proofs. In The Eleventh International Conference on Learning Representations , 2023.

Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Maxwell Forbes, Jon Borchardt, Jenny Liang, Oren Etzioni, Maarten Sap, and Yejin Choi. Delphi: Towards machine ethics and norms. CoRR , abs/2110.07574, 2021.

Philip N Johnson-Laird, Sangeet S Khemlani, and Geoffrey P Goodwin. Logic, probability, and human reasoning. Trends in cognitive sciences , 19(4):201-214, 2015.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR , abs/2001.08361, 2020.

Jon Kleinberg and Eva Tardos. Algorithm Design . Addison-Wesley Longman Publishing Co., Inc., USA, 2005.

Philipp E. Koralus and Vincent Wang-Mascianica. Humans in humans out: On GPT converging toward common sense in both success and failure. CoRR , abs/2303.17276, 2023.

Itay Levy, Ben Bogin, and Jonathan Berant. Diverse demonstrations improve in-context compositional generalization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1401-1422, Toronto, Canada, July 2023. Association for Computational Linguistics.

Zhengzhong Liang, Zeyu Zhang, Steven Bethard, and Mihai Surdeanu. Explainable verbal reasoner plus (evr+): A natural language reasoning framework that supports diverse compositional reasoning. arXiv preprint arXiv:2305.00061 , 2023.

Chu-Cheng Lin, Aaron Jaech, Xin Li, Matthew R. Gormley, and Jason Eisner. Limitations of autoregressive models and their alternatives. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 5147-5173, Online, June 2021. Association for Computational Linguistics.

Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 158-167, 2017.

Bingbin Liu, Jordan T Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Transformers learn shortcuts to automata. In The Eleventh International Conference on Learning Representations , 2022.

Bingbin Liu, Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Transformers learn shortcuts to automata. In International Conference on Learning Representations , 2023.

Ziming Liu, Ouail Kitouni, Niklas S Nolte, Eric Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. Advances in Neural Information Processing Systems , 35:34651-34663, 2022.

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. CoRR , abs/2303.17651, 2023.

William Merrill and Ashish Sabharwal. The expresssive power of transformers with chain of thought. arXiv preprint arXiv:2310.07923 , 2023.

William Merrill and Ashish Sabharwal. A logic for expressing log-precision transformers. Advances in Neural Information Processing Systems , 2023.

William Merrill and Ashish Sabharwal. The parallelism tradeoff: Limitations of log-precision transformers. Transactions of the Association for Computational Linguistics , 11:531-545, 2023.

William Merrill, Nikolaos Tsilivis, and Aman Shukla. A tale of two circuits: Grokking as competition of sparse and dense subnetworks. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models , 2023.

Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher Manning. Grokking of hierarchical structure in vanilla transformers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages 439-448, Toronto, Canada, July 2023. Association for Computational Linguistics.

Benjamin Newman, John Hewitt, Percy Liang, and Christopher D Manning. The eos decision and length extrapolation. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP , pages 276-291, 2020.

Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models, 2021.

Maxwell I. Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models. CoRR , abs/2112.00114, 2021.

OpenAI. ChatGPT: Optimizing language models for dialogue, 2022.

OpenAI. GPT-4 technical report, 2023.

Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177 , 2022.

Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations , 2022.

Patrick Prosser. Hybrid algorithms for the constraint satisfaction problem. Computational intelligence , 9(3):268-299, 1993.

Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is ChatGPT a general-purpose natural language processing task solver? CoRR , abs/2302.06476, 2023.

Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John F. J. Mellor, Irina Higgins, Antonia Creswell, Nathan McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, L. Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, N. K. Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Tobias Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d'Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew G. Johnson, Blake A. Hechtman, Laura Weidinger, Iason Gabriel, William S. Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem W. Ayoub, Jeff Stanway, L. L. Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training gopher. ArXiv , abs/2112.11446, 2021.

Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot numerical reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2022 , pages 840-854, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.

Maarten Sap, Ronan Le Bras, Daniel Fried, and Yejin Choi. Neural theory-of-mind? on the limits of social intelligence in large LMs. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 3762-3780, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.

Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought. In International Conference on Learning Representations , 2023.

Avi Schwarzschild, Eitan Borgnia, Arjun Gupta, Furong Huang, Uzi Vishkin, Micah Goldblum, and Tom Goldstein. Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks. In Advances in Neural Information Processing Systems , volume 34, pages 6695-6706. Curran Associates, Inc., 2021.

Herbert A. Simon. The architecture of complexity. Proceedings of the American Philosophical Society , 106(6):467-482, 1962.

Herbert A Simon and Allen Newell. Human problem solving: The state of the theory in 1970. American psychologist , 26(2):145, 1971.

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew K. Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakas, and et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. CoRR , abs/2206.04615, 2022.

Keith Stenning and Michiel Van Lambalgen. Human reasoning and cognitive science . MIT Press, 2012.

Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. Proofwriter: Generating implications, proofs, and abductive statements over natural language. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages 3621-3634, 2021.

Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. CoRR , abs/2211.09085, 2022.

Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Kathleen S. Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed H. Chi, and Quoc Le. Lamda: Language models for dialog applications. CoRR , abs/2201.08239, 2022.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models. CoRR , abs/2302.13971, 2023.

Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can't plan (a benchmark for LLMs on planning and reasoning about change). In NeurIPS 2022 Foundation Models for Decision Making Workshop , 2022.

Vikrant Varma, Rohin Shah, Zachary Kenton, János Kramár, and Ramana Kumar. Explaining grokking through circuit efficiency. arXiv preprint arXiv:2309.02390 , 2023.

Petar Veliˇ ckovi´ c and Charles Blundell. Neural algorithmic reasoning. Patterns , 2(7):100273, 2021.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research , 2022. Survey Certification.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems , volume 35, pages 24824-24837. Curran Associates, Inc., 2022.

Sean Welleck, Jiacheng Liu, Ximing Lu, Hannaneh Hajishirzi, and Yejin Choi. NaturalProver: Grounded mathematical proof generation with language models. In Advances in Neural Information Processing Systems , volume 35, pages 4913-4927. Curran Associates, Inc., 2022.

Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. In International Conference on Learning Representations , 2023.

Sean Welleck, Peter West, Jize Cao, and Yejin Choi. Symbolic brittleness in sequence models: on systematic generalization in symbolic mathematics. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 8629-8637, 2022.

Yi Zhang, Arturs Backurs, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, and Tal Wagner. Unveiling transformers with LEGO: a synthetic reasoning task. CoRR , abs/2206.04301, 2022.

Haoyi Zheng and Huichun Zhan. ChatGPT in scientific writing: a cautionary tale. The American Journal of Medicine , 2023.

Hattie Zhou, Azade Nova, Aaron Courville, Hugo Larochelle, Behnam Neyshabur, and Hanie Sedghi. Teaching algorithmic reasoning via in-context learning, 2023.

Appendices

A Compositional Tasks

A.1 Multiplication

Data Construction We exhaustively generate multiplication problems as question-answer pairs (e.g., Q: 'What is 4 times 32?' A: '128'). We focus on multiplications of two numbers x = ( x 1 , x 2 , . . . , x k ) and y = ( y 1 , y 2 , . . . , y k ) where each number can have up to k digits, amounting to 9 × 10 ( k -1) combinations per each number. We set k to 5 in our experiments. Figure 8 showcases an example prompt for performing few-shot learning without the inclusion of a scratchpad, while Figure 9 demonstrates an example prompt using a scratchpad. Throughout our experimentation, we explored various versions of the scratchpad, ranging from verbose and detailed to more concise alternatives. Among these variations, the scratchpad version depicted in Figure 9 ultimately produced the most favorable outcomes. Listing 1 shows the Python code for solving the task.

A.2 Einstein's Puzzle

Data Construction In our experiments, we initially establish a set of properties, such as Color, PhoneModel, Pet, and so forth, along with their corresponding values expressed in natural language templates (e.g., 'The house has a red color.'). We then devise a fundamental and straightforward set of clue types: 1) 'found_at', e.g., 'Alice lives in House 2', 2) 'same_house', e.g., 'The person who is a cat lover lives in the house that has a red color.', 3) 'direct_left', e.g., 'The person who has a dog as a pet lives to the left of the person who lives in a red house.', and 4) 'besides', e.g., 'The person who has a dog as a pet and the person who has a red house live next to each other.' In addition, we also set up harder clue types such as 'not_at', 'left_of' (not necessarily directly left of), 'two_house_between', etc. which are only used in auxiliary experiments.

The solution to the puzzle is a matrix of size K × M , where K represents the number of houses and M the number of attributes. During the puzzle generation, the M properties are randomly selected from the candidate pool, followed by the random sampling of K values for each property. The sampled values are then randomly permuted and assigned within the table to create the solution. It is important to note that we ensure one of the sampled properties is 'Name' to enhance the readability and comprehensibility of the puzzles. To construct the clues, we initially over-generate all valid clues based on the solution and subsequently remove redundant clues at random until we obtain a set with a

/negationslash

Algorithm 1

Puzzle Solver

Input:

Clues

Output:

Reasoning

path

1:

function

PuzzleSolver

(

Clues

)

2:

Path

[]

3:

LeftClues

clues

4:

while

|

LeftClues

|

= 0

do

5:

for

i=1 to

|

LeftClues

|

do

6:

CandidateClues =

(

|

LeftClues

|

i

)

7:

for

clue

in CandidateClues

do

8:

if

solve any cell

then

9:

LeftClues.remove(clue)

10:

Path.append(clue)

11:

return

Path

Reasoning Path Generation

General Unique Rules

There are 3 houses (numbered 1 on the left, 3 on the right). Each has a different person in them. They have different characteristics:

Each person has a unique name: peter, eric, arnold

People have different favorite sports: soccer, tennis, basketball

People own different car models: tesla, ford, camry

Clues

The person who owns a Ford is the person who loves tennis.

Arnold is in the third house.

The person who owns a Camry is directly left of the person who owns a Ford.

4.

Eric is the person who owns a Camry.

The person who loves basketball is Eric.

The person who loves tennis and the person who loves soccer are next to each other.

House

Name

Sports

Car

1

Basketball

Camry

2

Tennis

Ford

3

Soccer

Tesla

Ground-Truth Table

Clue #5

Clue #6

Clue #4

Name

Sports

Car

Basketball

Camry

Tennis

Soccer

<UniqueValues>

…

Name

Sports

Basketball

Tennis

Soccer

Name

Clue #2

unique solution, as previously sampled. This process ensures a coherent and engaging puzzle-solving experience. Refer to Figure 10 for an example.

Graph Construction Algorithm To solve the complex compositional reasoning process for a logical grid puzzle, we use existing puzzle solvers [21] to generate the computation graph. It follows the basic greedy principle of applying the minimum number of rules to solve any cell, i.e., if using only one rule to solve any given cell, then apply this rule. This algorithm iterates through all clues in the clue set until one or a set of clue combinations can solve any cell in the table. While it may not be the most efficient way to solve the puzzle, it provides models with explicit scratchpad verbalization through an intuitive computation graph. Refer to Figure 10 for the pseudo-code of the process, and Figure 11 for a scratchpad example.

A.3 Dynamic Programming Problem

A.3.1 Solution to this problem

Let a = [ a 1 , . . . , a n ] be an input. Let dp i be the maximum sum of a subsequence that does not include adjacent elements, when considering only the elements of the input from the i -th position onwards.

Trivially, dp n = max( a n , 0) since we only want to choose a number if it is non-negative. Moreover, dp n -1 = max( a n , a n -1 , 0) since we cannot choose adjacent numbers.

For any given dp i with i ≤ n -2 , we can express it in terms of dp i +1 and dp i +2 . Concretely, the maximum sum of a subsequence starting at position i may or may not include the element in the i -th position, a i . If the subsequence includes a i , then the maximum sum is a i + dp i +2 , since using a i blocks us from using the next element. If the subsequence does not include a i , then its sum is dp i +1 . Moreover, the answer may never be less than zero, because otherwise we would select the empty sequence 7 . In summary,

We now have a recursion with its base cases dp n = max( a n , 0) and dp n -1 = max( a n , a n -1 , 0) , and we can therefore compute all values in O ( n ) . It now only rests to reconstruct the lexicographically smallest subsequence that maximizes the desired sum, based solely on the computed dp values.

Starting from dp 1 and iterating sequentially through dp n -2 , we choose an item if and only if dp i = a i + dp i +2 (that is, the maximum sum comes from choosing the current element) and we have not chosen the previous element. This helps disambiguate cases where choosing or not choosing a i yields the same sum, but possibly only one of those will not incur in choosing adjacent numbers. Similarly, for positions i = n -1 and i = n we choose the element if dp i = a i (that is, choosing the element yields the maximum sum) and we have not chosen the immediately previous element. See an example Python solution in 2.

7 We don't need to explicitly check for this since dp n ≥ 0 . However, we include the condition to ease the scratchpad logic.

Data Construction We exhaustively generate data for this DP task. For question-answer setting, we include a thorough explanation of the task before asking to generate a solution (see Figure 12). We use all lists up to 5 elements as training, and we consider only lists where elements are in the range [ -5 , 5] (giving a total of 11 n lists for an input list of size n ). For out-of-domain evaluation, we use lists of sizes 6 to 10 inclusive. Example scratchpads and zero-shot prompts are shown in Figure 13 and 12 respectively. The scratchpad is generated automatically through templates. We considered five exemplars for the few-shot setup.

B Experimental Setups & Empirical Results

B.1 Models

For our experiments, we evaluate the performance of 6 LLMs: GPT4 ( gpt-4 ) [58], ChatGPT ( GPT3.5-turbo ) [57], GPT3 ( text-davinci-003 ) [11], FlanT5 [17] and LLaMa [75]. The evaluations were conducted from January 2023 to May 2023 using the OpenAI API. We perform fine-tuning on GPT3 ( text-davinci-003 ) for the three tasks, observing faster convergence when training on question-scratchpad pairs rather than question-answer pairs. For question-answer pairs fine-tuning, we train separately the model for {14, 12, 4} epochs for multiplication, puzzle, and DP respectively, saving the best model based on the validation set. Regarding training on question-scratchpad pairs, we train the model for {16, 8, 2} epochs for multiplication, puzzle, and DP. The batch size is set to approximately 0.2% of the number of examples in the training set. Generally, we observe that larger batch sizes tend to yield better results for larger datasets. For the learning rate multiplier, we experiment with values ranging from 0.02 to 0.2 to determine the optimal setting for achieving the best results and chose 0.2. During inference, we set nucleus sampling p to 0.7 and temperature to 1. For each task, we evaluate the performance of each model on 500 test examples.

B.2 Limits of Transformers in Zero- and Few-shot Settings

Figure 15, Figure 17 and Figure 20 show the zero-shot performance of GPT4, ChatGPT, LLaMA and FlanT5 on the three tasks. Overall, there is a notable decline in performance as the task complexity increases (measured by graph parallelism for multiplication and DP, and propagation steps for puzzles as shown in Figure14). The few-shot performance with question-answer pairs results in minimal improvement over the zero-shot setting as depicted in Figure 16 and Figure 20 for the multiplication and DP tasks. In contrast, the few-shot setting did not lead to any improvement in the puzzle task.

GPT4zero-shot(Multiplication)

GPT4zero-shot(DP)

GPT4zero-shot(Puzzle)

1.5

2

1.2

1.0

1.0

1

3.8

0.5

0.6

.4

0.0

0

3.2

0.0

-0.5

0.2

0

5

10

15

1.501.752.00

2.25

4

6

8

Graphparallelism

Graphparallelism

Propagationsteps(logscale)

1

2

3

4

5

1

2

3

4

5

1

1

0.99

0.99

0.85

0.55

0.98

0.51

0.11

0.01

0.98

0.26

0.02

0

0

ChatGPT zero-shot

1

2

3

4

5

1

2

3

4

5

1

1

0.88

0.97

0.41

0.09

0.85

0.28

0.04

0

0.76

0.1

0

0

0

GPT3 zero-shot

1

2

3

4

5

1

2

3

4

5

0.6

0.5

0.42

0.32

0.19

0.17

0.22

0

0

0

0

0

0

0

0

LLaMA zero-shot (13B)

1

2

3

4

5

1

2

3

4

5

0.96

0.41

0.05

0.15

0.02

0

0.15

0

0

0

0.07

0

0

0

0

FlanT5-XXL zero-shot (11B)

B.3 Limits of Transformers with question-answer Training

Figure 18 and Figure 21 show the performance of GPT3 finetuned on question-answer pairs. The model was trained on various splits, considering the problem size, depth, and width of the computation graph. Specifically, for the multiplication task, the model was fine-tuned on a range of multiplication problems, spanning from 1-digit by 1-digit multiplication to 4-digit by 2-digit multiplication amounting to 1.8M pairs. As for the puzzle task, the model was fine-tuned on puzzles of sizes ranging from 2x2 to 4x4 resulting in a total of 142k pairs. Additionally, for the DP task, the model was fine-tuned on problems with a sequence length of 5 resulting in 41K pairs. In an additional setup, we divided

1

2

3

4

5

1

2

3

4

5

1

1

0.98

0.97

0.88

0.56

0.93

0.76

0.18

0.03

0.95

0.42

0.07

0

0

GPT4 few-shot

1

2

3

4

5

1

2

3

4

5

1

1

0.99

0.99

0.82

0.58

0.94

0.45

0.08

0.01

0.96

0.28

0.02

0

0

ChatGPT few-shot

1

2

3

4

5

1

2

3

4

5

1

1

0.88

0.97

0.49

0.09

0.88

0.21

0.04

0

0.8

0.17

0.03

0

0

GPT3 few-shot

1

2

3

4

5

1

2

3

4

5

0.61

0.52

0.42

0.31

0.19

0.18

0.22

0

0

0

0

0

0

0

0

LLaMA few-shot (13B)

1

2

3

4

5

1

2

3

4

5

0.96

0.42

0.05

0.14

0.02

0

0.16

0

0

0

0.08

0

0

0

0

FlanT5-XXL few-shot (11B)

2

3

4

5

6

2

3

4

5

6

0.8

0.4

0.1

0.4

0

0.5

0

0.2

0

0

0.1

0

0

0

0

0

0

0

0

0

0

0

0

0

0

ChatGPT zero-shot (Puzzle)

2

3

4

5

6

2

3

4

5

6

0.6

0.5

0.2

0.1

0

0.5

0

0.2

0

0

0.1

0

0

0

0

0

0

0

0

0

0

0

0

0

0

GPT3 zero-shot (Puzzle)

2

3

4

5

6

2

3

4

5

6

0.44

0.25

0.09

0.08

0

0.1

0.02

0.03

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

LLaMa zero-shot (Puzzle)

2

3

4

5

6

2

3

4

5

6

0.41

0.22

0.02

0.02

0

0.04

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

FlanT5 zero-shot (Puzzle)

GPT3ftw/oscratchpad(Puzzle)

Puzzle w/o Scratchpad

Puzzle w/o Scratchpad

2

1

0.95

0.89

0.85

0.65

1.0

1.0

GPT3Fine-tuning

GPT3five-shot

GPT3 five-shot

3

0.8

0.7

0.55

0.5

0.45

0.8

0.8

4

0.8

0.6

0.4

0.3

0

ccura

0.6

n

0.6

0.4

0.4

5

0.1

0.1

0

Q

0

0.2

0.2

6

0

0

0

0

0

0.0

in-domain data

0.0

in-domain data

2

3

4

5

6

2

4

6

8

2

4

6

8

Number ofhouses

Numberofproperties

those datasets based on the depth and width of the computation graph for all the tasks and finetuned on different splits. The results indicate a lack of generalization for out-of-domain (OOD) examples while showcasing near-perfect performance for in-domain examples. One hypothesis on why the model exhibit such a poor generaliztion is tokenization. So we train GPT2-XL from scratch on up to 4x4 (90M data points), we assign each digit to one token and each math symbol as well. However, the performance is still low and GPT2-XL fails to answer correctly 3x3 test examples.

GPT3 finetuning cost We will discuss here the approximate cost of fine-tuning GPT3 for the multiplication task. When fine-tuning with question-answer pairs, each example typically consists of around 20 tokens, and 250 tokens for question-scratchpad pairs. The cost for utilizing the text-davinci-003 model amounts to $0.02 (USD) per 1,000 tokens. With this particular setup, the total number of training examples required for multiplication up to 5 digits by 5 digits reaches an astonishing figure of approximately 9.1 billion examples. Should we choose to fine-tune GPT3 for 4 epochs on question-answer pairs, the cost would amount to $12 million and $700 million for question-scratchpad training. For a more comprehensive breakdown of the cost per problem size, please refer to Table 1.

Multiplicationw/oScratchpad

Multiplicationw/oScratchpad

GPT3finetuned

GPT3finetuned

1.0

1.0

GPT3five-shot

GPT3 five-shot

0.8

GPT3zero-shot

0.8

GPT3zero-shot

0.6

0.6

0.4

0.4

0.2

0.2

0.0

in-domain data

0.0

in-domain data

0

5

10

15

20

0

5

10

15

20

Graph depth

Graph width

2

4

6

8

10

0.2

0.4

0.6

0.8

GPT4 (Dynamic Programming)

GPT4 zero-shot

GPT4 five-shot

2

4

6

8

10

0.0

0.1

0.2

0.3

0.4

0.5

0.6

ChatGPT (Dynamic Programming)

ChatGPT zero-shot

ChatGPT five-shot

2

4

6

8

10

0.0

0.1

0.2

0.3

0.4

0.5

GPT3 (Dynamic Programming)

GPT3 zero-shot

GPT3 five-shot

B.4 Limits of Transformers with Explicit Scratchpad Training

Figure 23, 24, 22 show the performance of GPT3 finetuned on different splits of the tasks using questionscratchpad pairs. Specifically, for the multiplication task, the model was fine-tuned on a range of multiplication problems, spanning from 1-digit by 1-digit multiplication to 3-digit by 2-digit multiplication.

As for the puzzle task, the model was fine-tuned on puzzles of sizes ranging from 2x2 to 4x4. Additionally, for the DP task, the model was fine-tuned on problems with a sequence length of 5. Furthermore, different data splits were considered, including variations based on the number of hours, number of properties, depth and width of the graph, and the number of digits in the multiplication output. On all tasks, we can see that the model fails to generalize to OOD data while achieving perfect accuracy on in-domain

GPT3 finetuned w.scratchpad

1

1.0

N

1

0.92

0.8

1

0.91

0.02

2

0.6

0.06

0.28

0

0

U

0.4

0.2

0

0

0

0

0

0.0

1

2

3

4

5

data, indicating that it cannot learn the underlying computational rules.

B.5 Limits of Transformers with Explicit Scratchpad Prompting

Figure 25 shows the results. GPT-4 exhibits an increase in few-shot accuracy in most problem sizes when using question-scratchpad pairs of few-shot examples across the three tasks. While its performance surpasses that of zero-shot and few-shot with question-answer pairs, it tends to decline as the complexity of the tasks increases. The same applies for the rest of the models.

DynamicProgrammingw/oScratchpad

DynamicProgrammingw/oScratchpad

DynamicProgrammingw/oScratchpad

GPT3finetuned

GPT3finetuned

GPT3 finetuned

1.0

1.0

1.0

GPT3 five-shot

GPT3 five-shot

GPT3 five-shot

0.8

GPT3zero-shot

0.8

GPT3 zero-shot

0.8

GPT3 zero-shot

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

in-domain data

0.0

in-domain data

0.0

in-domain data

2

4

6

8

10

10

15

20

25

30

10

15

20

25

30

35

Problem size

Depth

DynamicProgrammingw.Scratchpad

DynamicProgrammingw.Scratchpad

DynamicProgrammingw.Scratchpad

1.0

GPT3finetuned

GPT3five-shot

1.0

GPT3 five-shot

GPT3finetuned

1.0

GPT3 finetuned

GPT3 five-shot

0.8

GPT3 zero-shot

0.8

GPT3 zero-shot

0.8

GPT3 zero-shot

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

in-domain data

0.0

in-domain data

0.0

in-domain data

2

4

6

8

10

10

15

20

25

30

10

15

20

25

30

35

Problem size

Depth

d

Puzzlew.Scratchpad

1

1.0

1.0

GPT3five-shot

GPT3five-shot

0.55

0.8

0.8

Accurac

0.6

ccura

0.6

0

0.4

0.4

0

0.2

0.2

0

0.0

in-domain data

0.0

in-domain data

6

2

4

6

8

2

4

6

8

Number ofhouses

Numberofproperties

GPT4promptingw.scratchpad

GPT4promptingw.Scratchpad

GPT4 prompting w. scratchpad

(Multiplication)

(Puzzle)

(DP)

L

1

2

1

1

0.8

0.9

0.7

2

1

0.99

3

0.5

0.5

0.2

0.2

0.1

1

1

0.880.780.650.4

0.150.02

0

3

1

0.94

0.92

4

0.4

0.4

0.1

0.1

0.1

2

3

4

5

6

7

8

9

10

4

1

0.78

0.2

0.02

5

0.2

0

0

0.1

0

5

1

0.37

0

0

0

6

0

0.1

0.1

0

0

1

2

3

4

5

2

3

4

5

6

C Surface Patterns

C.1 Relative Information Gain Predictions for Multiplication

C.2 Empirical Surface Pattern Analysis for Multiplication with GPT4, ChatGPT and GPT3

1

2

3

4

5

1

2

3

4

5

1

1

1

1

1

1

1

1

0.99

0.98

1

0.99

0.98

0.97

0.97

1

2

3

4

5

1

2

3

4

5

1

1

1

0.99

1

1

0.99

0.94

1

1

1

0.98

0.99

1

1

Accuracy last digit

1

2

3

4

5

1

2

3

4

5

1

1

0.99

0.98

0.98

0.72

0.98

0.98

0.68

0.58

1

0.88

0.61

0.63

0.63

Accuracy last two digits

1

2

3

4

5

1

2

3

4

5

1

1

1

1

1

1

1

1

1

1

1

0.99

1

1

1

Accuracy # digits

1

2

3

4

5

1

2

3

4

5

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

Accuracy first digit

1

2

3

4

5

1

2

3

4

5

1

1

1

1

1

0.99

0.99

0.99

0.99

0.99

1

0.99

0.99

0.98

0.98

Accuracy first two digits

1

2

3

4

5

1

2

3

4

5

1

1

1

1

1

0.99

1

0.98

0.93

0.95

1

0.98

0.96

0.87

0.87

1

2

3

4

5

1

2

3

4

5

1

1

1

0.99

1

1

1

0.99

0.98

0.97

1

0.98

0.95

0.9

0.9

Accuracy last digit

1

2

3

4

5

1

2

3

4

5

1

1

0.99

1

0.93

0.91

1

0.89

0.88

0.87

1

0.88

0.83

0.8

0.8

Accuracy last two digits

1

2

3

4

5

1

2

3

4

5

1

1

1

1

1

1

1

1

1

1

1

0.99

1

1

1

Accuracy # digits

1

2

3

4

5

1

2

3

4

5

1

1

1

1

0.99

0.99

1

0.98

0.97

0.98

1

0.98

0.92

0.86

0.86

Accuracy first digit

1

2

3

4

5

1

2

3

4

5

1

1

1

1

0.96

0.97

1

0.94

0.95

0.94

1

0.95

0.91

0.84

0.84

Accuracy first two digits

1

2

3

4

5

1

2

3

4

5

1

1

1

1

1

1

1

1

0.97

0.98

1

0.97

0.99

0.98

0.98

1

2

3

4

5

1

2

3

4

5

1

1

0.99

0.99

0.99

1

0.99

0.99

0.96

1

1

0.9

0.98

1

0.86

Accuracy last digit

1

2

3

4

5

1

2

3

4

5

1

1

0.99

0.97

0.89

0.69

0.96

0.97

0.53

0.56

0.99

1

1

1

0.63

Accuracy last two digits

1

2

3

4

5

1

2

3

4

5

1

1

0.99

1

1

1

1

1

0.99

1

1

0.98

0.99

1

0.9

Accuracy # digits

1

2

3

4

5

1

2

3

4

5

1

1

1

1

1

1

1

1

0.99

0.99

1

0.98

1

0.98

1

Accuracy first digit

1

2

3

4

5

1

2

3

4

5

1

1

1

1

1

0.99

0.99

0.99

0.99

0.99

1

0.99

0.99

0.98

0.98

Accuracy first two digits

1

2

3

4

5

1

2

3

4

5

1

1

0.98

1

1

1

0.76

0.8

0.65

0.8

0.85

0.92

0.98

0.8

0.75

1

2

3

4

5

1

2

3

4

5

1

1

0.99

1

1

1

0.98

1

1

0.93

1

0.98

0.98

0.91

0.64

Accuracy last digit

1

2

3

4

5

1

2

3

4

5

1

1

0.94

1

0.99

1

0.96

1

1

0.94

0.43

0.53

0.17

0.18

0.1

Accuracy last two digits

1

2

3

4

5

1

2

3

4

5

1

1

0.99

1

1

1

0.12

0.54

0

0

0.02

0.02

0

0

0

Accuracy # digits

1

2

3

4

5

1

2

3

4

5

1

1

1

1

1

1

1

0.86

0.97

0.64

0.84

0.74

0.69

0.62

0.5

Accuracy first digit

1

2

3

4

5

1

2

3

4

5

1

1

1

1

1

0.5

0.32

0.69

0.1

0.01

0.43

0.31

0.16

0.08

0.12

Accuracy first two digits

C.3 Relative Information Gain Predictions for Dynamic Programming Task

Let a i be the i -th element of the input sequence, and let o i be the i -th element of the output sequence. As shown in Table 3, a i is a good predictor of o i , and this is especially true for a 1 and a n -1 , the first and last elements of the sequence. This matches the task intuition, since one would never pick an element a i < 0 and decrease the final sum (one may pick a i = 0 if it makes a lexicographically smaller output sequence).

a i weakly helps to predict its neighbors. The only case of this behavior with RelativeIG>0.1 is at the start of the sequence, where the first element helps predict the value of the second. This again matches intuition, since a very high a 1 indicates that with high probability o 2 will not be selected for the final subsequence.

Similar behaviors, but with higher relative information gains overall, are observed when analyzing triples of consecutive elements in the list. Table 4 shows that o i is highly predicted by ( a i -1 , a i , a i +1 ) . Moreover, o i is highly predicted by both ( a i -2 , a i -1 , a i ) and ( a i , a i +1 , a i +2 ) , with the former generally having higher scores than the latter. This again matches the task intuitions, since the value of the neighbors helps determine whether to select a number for the subsequence; and asking for the lexicographically smallest sequence biases the output subsequence to care more about the previous numbers rather than the following ones. We believe that this last point is the cause of the weakly predictive power of ( a i -3 , a i -2 , a i -1 ) to predict o i ; whereas ( a i +1 , a i +2 , a i +3 ) is not shown, since all the relative information gain values were below 0.1.

C.4 Empirical Surface Pattern Results for Dynamic Programming Task

We observe that all analyzed models match the Relative Information Gain prediction that o 1 (whether the first element goes into the output sequence or not) should be the easiest value to predict (see Figures 30, 31, and 32). However, since GPT3 often predicts shorter output sequences than the required size, the analysis of the predictive power of o n -1 is only done for GPT4. In GPT4, we observe that o n -1 is among the easiest values to predict as expected by Relative Information Gain.

Problem size (input list length)

0.70

0.80

0.90

1.00

2

4

6

8

10

o_1

o_2

o_3

o_4

Problem size (input list length)

0.70

0.80

0.90

1.00

2

4

6

8

10

o_{n-1}

o_{n-2}

o_{n-3}

o_{n-4}

o_{n-5}

Problem size (input list length)

0.60

0.70

0.80

0.90

1.00

2

4

6

8

10

Output size is correct

Model never chooses negatives

Model respects the non-adjacent condition

Problem size (input list length)

0.35

0.45

0.55

0.65

0.75

6

7

8

9

10

o_1

o_2

o_3

o_4

Problem size (input list length)

0.00

0.25

0.50

0.75

1.00

6

7

8

9

10

Output size is correct

Model never chooses negatives

Model respects the non-adjacent condition

Problem size (input list length)

0.7

0.8

0.9

1.0

6

7

8

9

o_1

o_2

o_3

o_4

Problem size (input list length)

0.00

0.25

0.50

0.75

1.00

6

7

8

9

Output size is correct

Model never chooses negatives

Model respects the non-adjacent condition

D Theoretical Results: Derivations

D.1 Error accumulates with larger parallel applications of an estimated function ( width )

Here we provide formal statements and derivations to Propositions 4.1 and 4.2 shown in the main paper. The mathematical framework used is a simplified representation of how multi-step reasoning works, showing two quintessential reasoning types: independent applications of the same step, or consecutive applications of the same step. We take an error estimation and accumulation perspective, since transformers are still being investigated from a theoretical standpoint.

̸

̂ ̂ Moreover, if c n ≤ βα n for some some α ∈ (0 , 1) and β > 0 , then lim n → + ∞ P ( f n = ̂ f n ) = 1 .

Proposition D.1. Let f n ( x ) = h n ( g ( x , 1) , g ( x , 2)) , . . . , g ( x , n )) . Let ̂ h n , ̂ g, ̂ f n be estimators of h n , g, f n respectively. Assume P ( h n = ̂ h n ) = 1 and P ( h n ( X ) = h n ( Y ) | X = Y ) < c n , where c n < c for some constant c ≪ 1 (i.e. ̂ h n perfectly estimates h n , and h n is almost injective). If P ( g = g ) = ϵ > 0 and errors in g are independent, lim inf n → + ∞ P ( f n = ̂ f n ) ≥ 1 -c .

̸

Proof. For ease of writing, let X i = g ( X,i ) and Y i = ̂ g ( X,i ) , and let X = ( X 1 , . . . , X n ) and Y = ( Y 1 , . . . , Y n ) . We will compute some auxiliary probabilities, and then upper bound P ( f = ̂ f ) , to finally compute its limit.

̸

̸

̸

̸

̸

̸

̸

which concludes our proof.

Note: In the case where c n ≤ βα n , we can derive an even stronger conclusion. In this case, we can prove that lim n → + ∞ P ( f n = ̂ f n ) = 1 . Recall that P ( f n = ̂ f n ) < αβ n +(1 -ϵ ) n · (1 -αβ n ) . Note that since 1 -ϵ ∈ [0 , 1) and α ∈ (0 , 1) , trivially lim n → + ∞ βα n +(1 -ϵ ) n · (1 -βα n ) = 0 .

̸

To conclude our proof, we will compute a lower bound for lim inf n → + ∞ P ( f n = ̂ f n ) . Note that since c n < c for all n , we know that P ( f n = ̂ f n ) < c +(1 -ϵ ) n · (1 -c ) . Then, P ( f n = ̂ f n ) > 1 -c -(1 -ϵ ) n · (1 -c ) . Since 1 -ϵ ∈ [0 , 1) , lim n → + ∞ 1 -c -(1 -ϵ ) n · (1 -c ) = 1 -c . Thus,

̸

̸

̸

̸

̸

̸

̸

Proof. We first derive a recursive upper bound using the law of total probability, and then prove a non-recursive upper bound by induction.

Then, lim n → + ∞ P ( f n = ̂ f n ) = 0 and we conclude lim n → + ∞ P ( f n = ̂ f n ) = 1 , assuming that c n ≤ βα n for some adequate α, β .

̸

Corollary D.1. Assume that a model M solves shifted addition perfectly, but it incorrectly solves at least one m digit by 1 digit multiplication for some fixed m . Then, the probability that M will solve any m digit by n digit multiplication using the long-form multiplication algorithm tends to 0 when n tends to infinity.

Proof. Let g = d ◦ s define the base-10 multiplication between m -digit numbers ( x 1 x 2 . . . x m ) and 1-digit numbers ( x m + j ), where s : Z m + n 10 × N → N × N denotes the selection of the numbers to multiply and d : N × Z 10 → N denotes the actual multiplication:

g := d ◦ s d ( x, y ) := x · y s ([ x 1 , . . . , x m , x m +1 , . . . , x m + n ] , j ) := ( x 1 + + x 2 + + . . . + + x m , x m + j ) where x 1 + + x 2 + + . . . + + x m denotes concatenating digits x i

Let h n : N n → N describe the shifted addition used at the end of long-form multiplication to combine n m -digit by 1 -digit multiplications, and let f n : Z m + n 10 → N describe the long-form multiplication of m -digit by n -digit numbers:

̸

̸

By hypothesis, P ( g = ̂ g ) = ϵ > 0 and P ( h n = ̂ h n ) = 1 , where ̂ g and ̂ h n denote estimators using model M . It can be shown that P ( h n ( X ) = h n ( Y ) | X = Y ) < βα n for α = 0 . 1 and β = 10 m . Using Lemma D.1, lim n → + ∞ P ( f n = ̂ f n ) = 1 , which concludes our proof.

̸

Note that Lemma D.1's proofs gives us empirical bounds once ϵ and α are approximated. Also note that our definition of g in the proof of Corollary D.1 highlights two possible sources of exponentially-accumulating error : errors in the selection of the numbers to multiply s , and errors in the actual m -digit by 1 -digit multiplication d .

D.2 Error accumulates with larger iterative applications of an estimated function ( depth )

̸

Proposition D.2. Let f n ( x ) = g n ( x ) . Assume P ( g ( X ) = ̂ g ( Y ) | X = Y ) ≤ c (i.e. recovering from a mistake due to the randomness of applying the estimator on an incorrect input has probability at most c ). If P ( g = ̂ g ) = ϵ > 0 with c + ϵ < 1 , then lim inf n → + ∞ P ( f n = ̂ f n ) ≥ 1 -c/ ( c + ϵ ) .

̸

̸

̸

We know s 1 = (1 -ϵ ) since s 1 = P ( f 1 = ̂ f 1 ) = P ( g = ̂ g ) . Let b := 1 -ϵ -c for ease of writing. Then, we have s n ≤ b · s n -1 + c (4)

It can be easily shown by induction that s n ≤ b n -1 (1 -ϵ ) + c ∑ n -2 i =0 b i :

̸

The base case n = 2 is true since we know s 2 ≤ b · s 1 + c , and b · s 1 + c = b (1 -ϵ ) + c = b 2 -1 (1 -ϵ ) + c ∑ 2 -2 i =0 b i , thus showing s 2 ≤ b 2 -1 (1 -ϵ ) + c ∑ 2 -2 i =0 b i · The inductive step yields directly using Equation 4,

We can rewrite the geometric series ∑ n -2 i =0 b i in its closed form 1 -b n -1 1 -b , and recalling b := 1 -ϵ -c ,

̸

Recalling that s n = P ( f n = ̂ f n ) , we compute the limit inferior of P ( f n = ̂ f n ) = 1 -s n ≥ 1 -b n -1 (1 -ϵ -c c + ϵ ) -c c + ϵ .

̸

that concludes our proof.

We can generalize the proof in Lemma 4.2 to tasks where there are potentially many valid reasoning chains with the following alternative state-transition framing.

Lemma D.2. Let S denote the set of all possible states a language model can generate, and let z : S →{ 0 , 1 } defines if a state is valid (0 = invalid). Let ̂ g : S → Π( S ) be a state-transition function representing a language model's probability distribution of generating each possible next state when attempting to perform a single reasoning step. Assume P ( z ( ̂ g ( X )) = 1 | z ( X ) = 0) ≤ c and P ( z ( g ( X )) = 0 | z ( X ) = 1) = ϵ > 0 with c + ϵ < 1 . Then, lim inf n → + ∞ P ( z ( g n ) = 0) = 1 -c/ ( c + ϵ ) .

̂ ̂ If for task T we know that all valid reasoning chains to arrive at a correct result have at least length n (i.e., the equivalent of defining f n = g n in Lemma D.1) then the probability of solving task T correctly tends to at most c/ ( c + ϵ ) .

Corollary D.3. The recursions for dynamic programming tasks, the m -by1 digit multiplication, and the puzzle's elimination function are all tasks where there is a fixed reasoning step g being repeatedly applied. Therefore, we can directly apply Proposition 4.2 to these tasks.

Proof. Let's analyze the three tasks separately below.

m -by1 digit multiplication may be viewed as f m ( x ) Let x = ( x 1 , . . . , x m ) be the m -digit number that we multiply by the 1-digit number y ( 0 ≤ y < 10 ). Let z = ( z 1 , . . . , z m +1 ) denote z = x · y , which is guaranteed to have exactly m +1 digits (with possibly leading zeros). We define f as:

where x ′ i := ( x i · y + c ) mod 10 and c ′ := ⌊ ( x i · y + c ) / 10 ⌋ . Note that x ′ i = z i +1 since f is performing one step of the long-form multiplication algorithm.

Let the initial input be x := ( x 1 , . . . , x m , y, m, 0) . Then, it can be easily shown that f m ( x ) = ( z 2 , . . . , z m +1 , y, 0 , c ) . Since c is the left-most carry, it is the leading digit of z , i.e. c = z 1 (possibly zero) . Thus, the value of z can be directly extracted from f m ( x ) = ( z 2 , . . . , z m +1 , y, 0 , z 1 ) .

In the DP task, dp 's computation may be viewed as f m -2 ( x ) for a list of size m See §A.3.1 for details on the solution to this problem. We will use identical notation. Let a 1 , . . . , a m be an input list. Let x = ( a 1 , . . . , a m -2 , a ′ m -1 , a ′ m , m -2) , where a ′ m := max( a m , 0) and a ′ m -1 := max( a m -1 , a m , 0) . Intuitively, this means that we have applied the first two steps of the dp computation, and stored the results in a ′ m -1 and a ′ m . Let f be a function representing the recursive computation of dp i :

where a ′ i := max( a ′ i +1 , a i + a ′ i +2 , 0) .

Note that since a ′ i +1 stores the value of dp i +1 and a ′ i +2 stores the value of dp i +2 , it can be easily shown that f m -2 ( x ) = ( a ′ 1 , . . . , a ′ m , 0) = ( dp 1 , . . . , dp m , 0) . Therefore, f m -2 computes all recursive values of dp i when given the base cases.

In the DP task, the reconstruction of the desired subsequence given already computed dp values may be viewed as f m ( x ) for an input list of size m . This case is similar to the previous one. Let r = ( r 1 , . . . , r m ) be the result, where r i = 1 if a i was selected for the desired subsequence, and r i = 2 otherwise. Let x := ( dp 1 , . . . , dp m , 0 , 0 , a 1 , . . . , a m , 1 , 1) . Let f be defined as follows:

where a ′ i := 2 -1 { dp i = a i + dp i +2 and u = 1 } and u := 1 -1 { dp i = a i + dp i +2 and u = 1 } . Intuitively, a ′ i stores whether the i -th element of the list should be selected for the final subsequence, assigning 1 if the element should be taken, and 2 otherwise (i.e., a ′ i = r i ). Moreover, if the i -th element has been selected, we mark that the next item will not be available using u ′ . Therefore, f performs one step of the final output reconstruction as defined in §A.3.1.

It can be easily shown that f m ( x ) := ( dp 1 , . . . , dp m , 0 , 0 , a ′ 1 , . . . , a ′ m , m + 1 , u ′ ) = ( dp 1 , . . . , dp m , 0 , 0 , r 1 , . . . , r m , m + 1 , u ′ ) . Note that the extra two elements in the input state allow lifting the special cases m -1 and m in the solution shown in §A.3.1 without falling out of bounds.

Solving the puzzle task may be seen as f m for some m , where f is the elimination function Let c 1 , . . . , c n be the list of clues, let H be the number of houses, and let A be a partially filled solution of size K × M as defined in §2.4. Each cell A ij can take H +1 values: the H options for the cell and the value ø, implying this cell has not been filled. An elimination step f may be defined as:

̸

where A ′ is also a partially filled matrix, with A ij = A ′ ij for every A ij = ø and where A ′ has at least one more filled cell.

Let x = ( c 1 , . . . , c n , E ) where E is an empty matrix of size K × M (all cell values of E are ø).

Then, a full solution is computed as f m ( x ) for some value of m that increases with the problem size. In contrast to other tasks, the value of m is not fixed, and depends on the task instance, but using solvers we know that m increases with problem size.

D.3 Discussing c ≪ ϵ in the context of Proposition 4.2

Note that in Proposition 4.2, if c ≪ ϵ then lim inf n → + ∞ P ( f n = ̂ f n ) ≈ 1 . This is because assuming ϵ = m · c for some m > 0 , we have 1 -c c + ϵ = 1 -c c + m · c = 1 -1 m +1 = m m +1 , and m m +1 is a monotonically increasing function for all m> 0 that tends to 1 when m goes to infinity. Therefore, large m 's (or alternatively, c ≪ ϵ ) imply m m +1 will be close to 1.

̸

It is reasonable to assume c ≪ ϵ when g has low collision, since c represents the probability of the estimator g ( y ) arriving at the correct output g ( x ) by chance when given the wrong input y = x .

̂ If g is discrete, it can take | Im ( g ) | values, where | Im ( g ) | denotes the cardinal of the image space of g . Assuming approximately uniform errors, c ≈ ϵ/ | Im ( g ) | , which in turn implies c ≪ ϵ since g being low collision implies | Im ( g ) | is large.

̸

If g is continuous, under appropriate assumptions it seems plausible that we can prove that c ≈ 0 (e.g. if errors are approximately uniform).

̸

Summarizing both cases, if errors are approximately evenly distributed we obtain that lim inf n → + ∞ P ( f n = ̂ f n ) ≈ 1 .

E Additional Literature and Societal Impact

E.1 Additional Literature

Iterated Functions The process of repeatedly applying a noisy single operation or function f can be related to iterated random functions [24]. In this latter literature, the focus is usually on the contractive regime in which accrued errors can be kept under control, and the subsequent convergence guarantees (e.g., [23]). When f is an affine transformation, the process falls simultaneously between two perspectives: time series [31] and dynamic programming and control [6]. We leverage the former to discuss the often explosive errors of f n .

E.2 Societal Impact Discussion

Our work on analyzing the limitations of current transformers in compositional tasks can have a positive societal impact in several ways. By shedding light on these limitations, we contribute to a deeper understanding of the capabilities and constraints of these models. This knowledge is essential for researchers, developers, and policymakers in making informed decisions regarding the application of transformers in various domains.

Understanding the limitations of transformers in compositional reasoning is crucial for developing more reliable and robust AI systems. By identifying these shortcomings, we can direct future research efforts toward addressing these limitations and developing models that exhibit improved performance in handling complex tasks requiring compositional reasoning.

We do not foresee any negative societal impacts, as our analysis aims to understand the reasons behind transformers' failures and successes, but does not introduce any new model or dataset that future work may leverage.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the compositional reasoning capability of Transformer models on three kinds of tasks, namely multi-digit multiplication, logic grid puzzles, and a dynamic programming problem. The compositional complexity of a given task, or the number of reasoning steps, can be easily controlled by varying lengths of inputs. This paper formulates the reasoning steps as a computation graph and uses characteristics like lengths and widths of the computation graph to qualify the level of complexity. For experimental evaluation, this work evaluates several Transformer models (i.e., GPT3, ChatGPT, GPT-4) using zero-shot, few-shot, and finetuning techniques. This paper shows that in zero-shot and few-shot settings, all three Transformer models deteriorate sharply from nearly perfect to nearly zero when increasing compositional complexity. Furthermore, finetuning GPT-3 with or without scratchpads does not help GPT-3 generalize to unseen and more complicated task instances. To gain a deeper understanding of these failures, this work finds that 1) Transformer models are likely to give partially correct predictions on outputs that can be determined by a small set of input features, and 2) Transformer models are likely to rely on computation patterns seen in training data and generalize poorly when they are not sufficient for OOD tests. Finally, with reasonable assumptions, this work gives a theoretical justification for why any models fail to generalize longer compositional reasoning tasks. Informally, as long as every single step is not perfect, the error will happen for a sufficiently long reasoning chain.\n\nSTRENGTHS:\n- This work carefully selects three representative compositional tasks and introduces computation graphs to systematically quantify the level of complexity. \n- State-of-the-art transformer models like GPT-3, ChatGPT, and GPT-4 are used in the experimental evaluations. The observed limits are likely to be universal for existing transformer models. \n- Some new insights in analyzing the successes and failures of Transformer models are presented. Specifically, relative information gain seems a likely reason that Transformer models tend to predict partially correct answers; and linearized subgraph matching suggests Transformer models tend to capture spurious patterns in the training.\n- Four interesting categories of errors are proposed and can effectively characterize the mistaken behaviors of Transformer models when the compositional complexity increases. The restoration error is particularly interesting, as it captures an often ignored problem -- the model could give correct answers but because of wrong reasons. \n- Some simple but formal analyses are proposed to illustrate the theoretical limits of Transformer models.\n\nWEAKNESSES:\n- It is well-known that large language models like GPT-3, ChatGPT, and GPT-4 behave poorly on complicated reasoning and planning tasks.  So, most results of this work are somewhat expected.\n\n- Only three tasks are evaluated, and only one specific algorithm is chosen for each task. Although the chosen algorithm seems straightforward, it is possible that the particular algorithm may not be appropriate for Transformer models to learn. It does not rule out that the Transformer models may be good at learning other less intuitive algorithms for the evaluated tasks. \n\n- Finetuning is only performed on GPT-3. It is understandable that finetuning larger models like ChatGPT or GPT4 is computationally prohibitive for a small research group. But, from a scientific point of view, the observations on GPT-3 may not generalize to ChatGPT or GPT4, and the emergent capabilities of larger language models may overcome the limits observed on GPT-3.\n\n- The claim of subsection 3.2.2 is a bit subjective. Linearized subgraph matching is a means proposed in this work to better understand Transformer models. It does shine some light on why Transformer models fail for the chosen tasks, however, whether that's indeed Transformers' internal dynamics is not quite clear. \n\n- The theoretical justification seems oversimplified. It has nothing to do with the Transformer models.\n\nQUESTIONS:\nEven assuming the computation graph is unique for a specific problem instance, exponentially many different sequential steps could express the same computation. How many sequences are extracted for a problem instance? \n\nThe definition (line 213) of whether a full computation is seen during training is not quite clear, particularly what's the precise meaning of the sign $\\equiv$ here?\n\nFigure 5b shows that predictions with large graph depth (e.g., > 16) tend to be all correct and have zero frequencies. It seems the Transformer models do generalize to more complicated tasks without reusing seen computation patterns in the training. Is it a plotting issue? Or is this interpretation wrong for some reason?\n\nLIMITATIONS:\nThe authors discussed potential limitations, and there is no (obvious) negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the ability of LLMs for compositional tasks. They propose a formalization called computation graphs. With this formalization, they further propose three different tasks: multi-digit multiplication, Einstein’s puzzle, and a dynamic programming problem. They test multiple LLMs on these three tasks with different training and inference settings. By varying the depth and width of the graph, they show that LLMs do not do well when the scale of the problem is large. In their further analysis, they point out two problems of LLM models: relying on single-step shortcuts and relying on similar training subgraphs. They also conduct an error classification and notice many hints for memorization (e.g., restoration error). Finally, this work presents a theoretical analysis showing the effect of error propagation.\n\nSTRENGTHS:\n1.\tThis work presents a relatively comprehensive and detailed evaluation of the compositional abilities for LLMs. The empirical results contain multiple training settings, and results w/wo scratchpads. \n2.\tThe three tasks designed in this work can be valuable for future work on compositional tasks.\n3.\tThe error analyses in this work are detailed and insightful. I really like the error classification analysis, which shows a more detailed description about LLM behaviors, and provide concrete and quantitative evidence for memorization, restoration error and error propagation.\n\nWEAKNESSES:\n1.\tWhile the theoretical derivation is interesting, I find the takeaways and the perspective to be slightly underwhelming. The main takeaway seems to be error propagation lead to bad results when the problem scale is sufficiently large. This is true, but this is also true for everything, and for both machines and humans. To me, the more interesting problem lies in that why even n is not very large, compositional problems still seem to be relatively hard. Additionally, I think the theoretical derivation applies to sampling but not to greedy decoding. \n2.\tThe idea of using a computation graph to control and analyze the difficulty of composition is not novel, see related works in [1][2][3]. While I do feel that the approach and objective of this paper are different enough compared to previous works, there should still be a discussion about these works.\n3.\tMinor presentation issue: Some experimental setup details are hard to find. For example, I cannot find how large the finetuning datasets are. And some of the other important details (e.g., how you split the train/test for finetuning experiments) are in the appendix. I understand the space is tight in the main paper, but I would encourage the authors to add some details to the main paper so that the results can be interpreted more easily.\n\n\n\n[1] Josef Valvoda, Naomi Saphra, Jonathan Rawski, Adina Williams, and Ryan Cotterell. Benchmarking Compositionality with Formal Languages. In COLING 2022.\n\n[2] Keysers, Daniel, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev et al. Measuring Compositional Generalization: A Comprehensive Method on Realistic Data. In ICLR 2020.\n\n[3] Ben Bogin, Shivanshu Gupta, and Jonathan Berant. Unobserved Local Structures Make Compositional Generalization Hard. In EMNLP 2022.\n\nQUESTIONS:\n1.\tThe prompt in the scratchpad seems to be carefully designed. How many different prompts have you tried? How do different prompts influence the findings?\n2.\tHow do you compute the entropies used in the information gains?\n3.\tJust to confirm, when doing the question-answer training, is there no scratchpads?\n\nLIMITATIONS:\nThis work provided a limitation section in Sec. 8 in the main paper and a societal impact statement in Appendix E.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper contributes to the literature on analyzing the limitations of Transformer models in compositional tasks by presenting an analysis based on computation graphs. Based on these computation graphs, some interesting new observations are made, which I believe are worth publishing.\n\nI found the paper a great read and very interesting. I have many comments/criticisms/questions below, but I would like to clarify that these are not attempts to attack the content, and more aiming for constructive criticisms for trying to make the paper more clear and situated in the literature, as I think the work is very solid.\n\nSome typos/comments:\n- Part of this work has already been studied in the past, and I think the authors should do a better job in differentiating the new insights from previously known parts. For example, I think it'd be great to discuss the new findings given: \"Making transformers solve compositional tasks\" (2021), \"The devil is in the detail: Simple tricks improve systematic generalization of transformers\" (2021), and \"Grokking: Generalization beyond Overfitting on Small Algorithmic Datasets\" (2022)\n- Page 2: \"This substantial gap suggests that systematic problem-solving capabilities do not emerge from maximum likelihood training\" -> See the Grokking paper I mentioned above, as they show that maybe systematic problem-solving abilities can improve when training beyond the overfitting threshold.\n- Section 3: As discussed in \"Making transformers solve...\" mentioned above, even if \"vanilla\" Transformers really struggle to solve compositional tasks, slight modifications to the architecture show significant gains. So, when only evaluating variants of the GPT architecture, it seems like conclusions are limited to GPT-like models, and not to \"Transformers\" in general, as small architectural decisions can have big impact in these types of tasks. For example GPT models being decoder-only models, they employ causal attention, where as encoder-decoder models would be able to deploy bidirectional attention to the input problem, which can give very different inductive biases. Carefully clarifying the extent of the conclusions would be better.\n- Section 3.1: \"suggesting that systematic problem-solving capabilities do not emerge via exhaustive training on task-specific data.\" -> Again, I'd like to bring up the Grokking paper results, and I wonder if the authors could comment on the implications for their results here.\n- Section 3.1: Although results are very interesting, I think the authors over-generalize the conclusions from experiments with one particular model pretrained in one particular way. I think it might be good to constrain conclusions of results to decoder-only-GPT-style-pretrained-Transformer models.\n- Propositon 4.1: this has very little to do with Transformers, but with any method that approximates steps. And btw, this proposition applies to humans as well, as the probability that a human makes a long calculation correct decreases with the length of the calculation due to the growing probability that at least one of the steps is wrong. So, again, I think that clarifying the extent of the conclusions in the paper needs some work, whereas above I was pointing out that experiments show failure cases for GPT-style models and you are overgeneralizing to Transformers, here you are showing a limitation of sequential prediction models, and too narrowly just blaming Transformers. Moreover, I see this proposition very related to the classic error compounding in the literature of imitation learning (e.g., see \"Efficient Reductions for Imitation Learning\", 2010), and usual solutions, e.g. Dagger, rely on training models to recover from mistakes, mitigating the problem. It'd be great to comment on the relation if the authors see it appropriate.\n\nSTRENGTHS:\nEven if the weaknesses of Transformers in compositional tasks have been studied many times in the recent past, this paper brings an interesting new perspective based on computation graphs that allows for a more finegrained analysis than previous work.\n\nWEAKNESSES:\nPerhaps the two main weaknesses (fixable) are that (1) conclusions seem to be overgeneralizing to the whole class of Transformer models, when only one instance of them was used for evaluation, and (2) it seems that some of the most relevant pieces of work in the literature were not discussed, making it harder to see what are the new pieces of knowledge that this paper is bringing compared to what was already known. Both of them fixable with small modifications to the manuscript though.\n\nQUESTIONS:\n- In Section 3.2.3: How were these errors analyzed? Was the text output of the models parsed to find the intermediate steps? Assuming many of these are in natural language, how accurately were you able to recover these intermediate steps?\n- Concerning memorization/shortcuts: We humans use memorization and shortcuts all the time for performing some of these operations (e.g., we know that X times 0 is 0, no matter how many digits does X have, and hence we will skip all the computation graph, etc.). The key it seems to me is to know when these shortcuts are applicable, and when they are dangerous. Did you see any case of the former (correct application of shortcuts)? (apologies if this is explained somewhere in the paper or appendix, but I was unable to find it).\n- As a curiosity, where does the \"faith and fate\" from the title come from, it might be a lack of familiarity with some English idioms, but I am struggling to make the connection with the paper.\n\nLIMITATIONS:\nAs acknowledged by the authors in their own section, the main limitation is that experiments were conducted with just GPT models, and it's unclear if we can generalize from those, given these are all the same Transformer architecture, and pretrained in a particular way.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper develops empirical and theoretical studies on the limitations of Transformers for solving compositional tasks. It defines computation graphs and quantifies compositional complexity with the graph metrics and uses relative information gain to predict surface patterns learned by Transformer. It conducts extensive experiments on three representative compositional tasks and finds that Transformers rely on pattern matching and exhibits poor generalization in compositionality.

STRENGTHS:
This paper conducts extensive experiments and profound analysis. The idea of quantifying compositional complexity with computational graphs is inspiring.

WEAKNESSES:
The empirical findings and the driven conclusions are quite trivial.

QUESTIONS:
1. Based on the findings, what could be a possible solution for improving Transformers in multi-step compositional problems? 
2. Can the computational graph definition and graph metrics be generalized to other compositional tasks, for example geometry problems?

LIMITATIONS:
The paper lacks methodologies or instructions for improving the model based on the obtained empirical findings and driven conclusions.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06