SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Faith and Fate:
Limits of Transformers on Compositionality
Nouha Dziri $^{1*}$ , Ximing Lu $^{1,2*}$ , Melanie Sclar $^{2*}$ , Xiang Lorraine Li $^{1\dagger}$ , Liwei Jiang $^{1,2\dagger}$ , Bill Yuchen Lin $^{1\dagger}$ , Peter West $^{1,2}$ , Chandra Bhagavatula $^{1}$ , Ronan Le Bras $^{1}$ , Jena D. Hwang $^{1}$ , Soumya Sanyal $^{3}$ , Sean Welleck $^{1,2}$ , Xiang Ren $^{1,3}$ , Allyson Ettinger $^{1,4}$ , Zaid Harchaoui $^{1,2}$ , Yejin Choi $^{1,2}$
$^{1}$ Allen Institute for Artificial Intelligence  $^{2}$ University of Washington
<sup>3</sup>University of Southern California <sup>4</sup>University of Chicago
nouhad@allenai.org, ximinglu@allenai.org, msclar@cs.washington.edu
Transformer large language models (LLMs) have sparked admiration for their exceptional performance on tasks that demand intricate multi-step reasoning. Yet, these models simultaneously show failures on surprisingly trivial problems. This begs the question: Are these errors incidental, or do they signal more substantial limitations? In an attempt to demystify transformer LLMs, we investigate the limits of these models across three representative compositional tasks—multi-digit multiplication, logic grid puzzles, and a classic dynamic programming problem. These tasks require breaking problems down into sub-steps and synthesizing these steps into a precise answer. We formulate compositional tasks as computation graphs to systematically quantify the level of complexity, and break down reasoning steps into intermediate sub-procedures. Our empirical findings suggest that transformer LLMs solve compositional tasks by reducing multi-step compositional reasoning into linearized subgraph matching, without necessarily developing systematic problem-solving skills. To round off our empirical study, we provide theoretical arguments on abstract multi-step reasoning problems that highlight how autoregressive generations' performance can rapidly decay with increased task complexity.
"It was the epoch of belief, it was the epoch of incredulity." - Charles Dickens, A Tale of Two Cities
Large-scale transformers such as ChatGPT [57] and GPT4 [58] demonstrate unprecedented capabilities [57, 74, 11, 15, 85], even noted as “sparks of AGI” [12]. In stark contrast, the same models sometimes struggle with simple, intuitive tasks [9, 62, 40]. For instance, humans can solve 3-digit by 3-digit multiplication arithmetic after learning basic calculation rules [22, 34]. Yet, off-the-shelf ChatGPT and GPT4 achieve only  $55\%$  and  $59\%$  accuracies on this task, respectively (§3).
The striking discrepancy between the impressive successes of transformer LLMs on seemingly complex tasks and the astonishing failures on seemingly trivial tasks spark critical open questions about how to faithfully interpret their mixed capabilities. Under what conditions do transformers succeed, fail, and why? What types of errors do they make? Can transformers uncover implicit problem-solving rules or be taught to follow reasoning paths?
Seeking thorough answers to these questions remains an open research challenge. However, we offer novel insights into the fundamental limits of transformers<sup>2</sup>, centered around compositional
* First co-authors. † Second co-authors.
2For brevity, we use 'transformers' to refer to autoregressive transformer LLMs' throughout the paper.
problems that require strict multi-hop reasoning to derive correct predictions. Applying step-by-step reasoning is fundamental to human intelligence [69, 68]. These compositional problems present compelling challenges for AI systems as they require combining basic reasoning operations to follow computational paths that arrive at unique correct solutions. In particular, we study three straightforward and flexible representative compositional tasks: long-form multiplication, logic grid puzzles (i.e., Einstein's puzzle [61]), and a classic dynamic programming problem.
We propose two hypotheses. First, transformers solve compositional tasks by reducing multi-step compositional reasoning into linearized path matching. This contrasts with the systematic multi-step reasoning approach that learns to apply underlying computational rules required for building correct answers [71, 37, 27]. Shortcut learning [29] via pattern-matching may yield fast correct answers when similar compositional patterns are available during training but does not allow for robust generalization to uncommon or complex examples. Second, due to error propagation, transformers may have inherent limitations on solving high-complexity compositional tasks that exhibit novel patterns. Errors in the early stages of the computational process can lead to substantial compounding errors in subsequent steps, preventing models from finding correct solutions.
To investigate our hypotheses, we formulate compositional tasks as computation graphs. These graphs break down problem-solving into submodular functional steps, enabling structured measurements of complexity and verbalization of computational steps as input sequences to language models. Moreover, we leverage information gain to predict patterns that models are likely to learn based on the underlying task distribution without the need to perform full computations within the graph.
Empirical results show that training on task-specific data leads to near-perfect performance on indomain instances and under low compositional complexity, but fails drastically on instances outside of this region. This substantial gap suggests that systematic problem-solving capabilities do not emerge from maximum likelihood training [5] on input-output sequences, even when prompted or trained with human-like reasoning steps (i.e., a linearization of computation graphs; §3.1). Models' success can be attributed, in part, to their exposure to training examples sub-graphs that involve the same computations required for solving test examples (see Section 3.2.2) In order to gain a deeper understanding of models' failures, we conduct a comprehensive analysis by decomposing their computation graphs and examining different error types. We find that while models can memorize single-step operations, they fail to compose them into correct reasoning paths, suggesting that they mostly make predictions based on shallow, rote learning rather than a deep, holistic task understanding (§3.2.3). Importantly, we provide theoretical evidence of exponential error accumulation using abstract compositional tasks. All tasks analyzed empirically in this paper are instantiations of these abstractions (§4). We argue that transformers could be inherently limited in solving compositionally complex tasks out-of-the-box<sup>3</sup>.
As transformers continue to make tangible real-world impacts, it is pressing to interpret their remarkable performance critically. Our work takes a realistic look at the limitations of transformers in the context of compositional tasks. To shed light on practical future steps, we identify directions for addressing these limitations, such as using transformers for tasks that could be decomposed into few reasoning steps, tasks where evaluation may afford some leniency, and using transformers in combination with planning modules or refinement methods to improve their generations. To advance language AI, fundamental innovations are required to address or complement these limitations.
2 Measuring Limitations of Transformers in Compositional Tasks
Human problem-solving skills can be conceptualized as a graph structure, where each vertex represents a partial solution and the edges represent operators that can be applied to modify these solutions. As we will outline next and illustrate in Figure 1, we use computation graphs and corresponding metrics to methodically evaluate transformers' reasoning abilities.
2.1 Computation Graph Definition
Let  $A$  be a deterministic algorithm (function), and let  $\mathcal{F}_A$  be a set of primitives (functions) the algorithm uses in its execution. Assuming the inputs  $\mathbf{x}$  to algorithm  $A$  are given, we define  $A(\mathbf{x})$ 's static computation graph  $G_{A(\mathbf{x})}$ .  $G_{A(\mathbf{x})} = (V, E, s, op)$  is a directed acyclic graph. Nodes  $V$
3Code and data are available at https://github.com/nouhadziri/faith-and-fate
function multiply (x[1..p], y[1..q]): // multiply x for each y[i]  
for i = q to 1  
    carry = 0  
    for j = p to 1  
        t = x[j] * y[i]  
        t += carry  
            carry = t // 10  
            digits[j] = t mod 10  
summands[i] = digits  
// add partial results (computation not shown)  
product =  $\sum_{i=1}^{q} \text{summands[q+1-i]} \cdot 10^{i-1}$   
return product
Figure 1: Transformation of an algorithm  $A$  to its computational graph  $G_{A(\mathbf{x})}$ . The depicted example is of long-form multiplication algorithm  $A$ , for inputs  $\mathbf{x} = [7,49]$  (i.e. computing  $7 \times 49$ ).
represent all variables' values during  $A$ 's execution: each node  $v \in V$  has a value  $s(v) \in \mathbb{R}$  associated. Edges  $E$  represent the function arguments involved in some computation: for each non-source node  $v \in V$ , let  $U = \{u_1, \dots, u_j\} \subset V^j$  be its parent nodes. Then,  $s(v) = f(u_1, \dots, u_j)$  for some  $f \in \mathcal{F}_A$ . Since each node  $v$  is uniquely defined by the computation of a single primitive  $f$ , we define  $op : V \to \mathcal{F}_A$  as  $op(v) = f$ .
Let  $S \subset V$  be the source nodes of  $G_{A(\mathbf{x})}$  and without loss of generality, let  $o \in V$  be its sole leaf node. By definition,  $S \equiv \mathbf{x}$  and  $A(\mathbf{x}) = s(o)$ , representing the input and output of  $A$  respectively.
To be able to train and evaluate a language model's ability to follow algorithm  $A$  we must linearize  $G_{A(\mathbf{x})}$ . Since we only consider autoregressive models, this linearization must also be a topological ordering.
2.2 Quantifying Compositional Complexity using Graph Metrics
$A$  's representation as a computation graph  $G_{A(\mathbf{x})}$  enables measuring task complexity from many angles.
We define a node  $v \in V$ 's layer number as the length of the longest path from a source node to  $v$  in the directed acyclic graph  $G_{A(\mathbf{c})}$ . We then define the reasoning depth as the largest layer number in the graph. In computation graphs, reasoning depth is a proxy for the maximum level of multi-hop reasoning required to solve the task.
Let  $d_S: V \to \mathbb{N}_0$  be the shortest distance to any of  $G$ 's source nodes  $S \subset V$ . We define the reasoning width of a graph as the mode of  $\{d(v): v \in V\}$ . This metric aims to measure the maximum number of variables required to maintain in parallel during the computation. Relatedly, we also define the average parallelism of a graph as the ratio between  $|V|$  and its reasoning depth. This aims to compute the average width in computation through the graph, and not just in its mode.
2.3 Predicting Surface Patterns through Relative Information Gain
When evaluating model performance, we may observe partially correct answers even in an overall incorrect response. To understand model strategies in these partial successes, we use Relative Information Gain to predict surface patterns that models are likely to recognize. We represent task  $T$  as a distribution  $(X_{1},\ldots ,X_{n},Y_{1},\ldots ,Y_{m})$  and measure the amount of (normalized) information gained about an output element  $Y_{j}$  by observing a subset of input random variables  $X\subset \{X_1,\dots ,X_n\}$ :
$$
\operatorname {R e l a t i v e} \mathrm {I G} (Y _ {j}, X) = \frac {H (Y _ {j}) - H (Y _ {j} | X)}{H (Y _ {j})} \in [ 0, 1 ] \tag {1}
$$
RelativeIG may be used to analyze the influence of any node in the computation graph (as defined in §2.1) with respect to a set of its ancestors; in particular, output nodes with respect to input nodes.
2.4 Exploring Three Representative Compositional Tasks: Definitions
Multiplication Multi-digit multiplication requires executing operations with numerical symbols based on procedural rules [34]. This task has multiple algorithmic solutions; in constructing computation graphs, we use the well-known  $O(k_{1}k_{2})$  long-form multiplication algorithm for computing  $x\cdot y$ , where  $x$  has  $k_{1}\leq 5$  digits and  $y$  has  $k_{2}\leq 5$  digits in base 10. See §A.1 for data construction details.
Figure 2: (a) Zero-shot accuracy. Axes refer to problem sizes (number of digits in multiplication, number of houses and attributes in puzzle, and sequence length in the DP task). Transformers' accuracy decreases to near zero as task complexity increases, measuring task complexity by the problem size. (b) Average parallelism negatively correlates with accuracy.
To instantiate  $G_{A(\mathbf{x})}$ , let  $\mathcal{F}_A = \{\text{one-digit multiplication, sum, mod } 10, \text{ carry over, concatenation}\}$ . Source nodes  $S$  are digits of input numbers, leaf node  $o$  is the final output, and intermediate nodes  $v$  are partial results generated during execution of the long-form multiplication algorithm (see Figure 1).
Einstein's Puzzle Einstein's puzzle is a well-known logic puzzle often used as a benchmark for solving constraint satisfaction problems [61]. It involves a list of houses with different attributes (e.g., owner's name, pets), and the goal is to determine which attributes belong to each house by combining a set of pre-defined natural language clues or constraints. The solution to the puzzle is a matrix of size  $K \times M$ , where  $K$  represents the number of houses and  $M$  the number of attributes. As  $K$  and  $M$  increase, synthesizing different partial solutions that satisfy individual constraints becomes highly compositionally complex. To construct the computation graph, we consider a greedy algorithm that iteratively eliminates possible solutions by filling at least one cell each time. It deterministically fills the cell(s) that requires the minimum number of clues among all current unfilled cells. We refer to this as the elimination function. See §A.2 for examples, data construction, and algorithm details.
To instantiate  $G_{A(\mathbf{x})}$ , let  $\mathcal{F}_A = \{\text{elimination function}\}$ . The source nodes are the clues, all intermediate nodes are partially-filled matrices, and the output node is a fully-filled solution matrix.
Dynamic Programming Problem Dynamic programming (DP) recursively breaks down complex problems into simpler sub-problems, so problems solved using this technique are compositional. We analyze a classic relaxation of the NP-complete Maximum Weighted Independent Set problem [39]: Given a sequence of integers, find a subsequence with the highest sum, such that no two numbers in the subsequence are adjacent in the original sequence. This relaxation may be solved in  $O(n)$  time using DP. See the solution in §A.3. In the experiments, we restrict each integer to the  $[-5, 5]$  range.
To instantiate  $G_{A(\mathbf{x})}$ , let  $\mathcal{F}_A = \{\text{equals, and, not, indicator function, sum, max}\}$ . Source nodes are elements of the input list, and the output node is a list that for each element indicates whether it should be selected. We select an  $O(n)$  algorithm since  $G_{A(\mathbf{x})}$ 's size is proportional to  $A$ 's complexity.
3 Testing the Limits of Transformers: Empirical Evidence
Experimental Setup To understand the capabilities of LLMs, we evaluate GPT3 (text-davinci-003) [11], ChatGPT (GPT-3.5-turbo) [57] and GPT4 (gpt-4) [58] using zero-shot, few-shot, and finetuning techniques. To enable the generation of computation graphs beyond the final answers, we use the concept of scratchpads [56]. Scratchpads are a verbalization of the computation graphs (i.e., a linearized representation of a topological ordering of  $G_{A(\mathbf{x})}$ ). Overall, we consider question-answer and question-scratchpad formats for few-shot and finetuning settings to gauge models' capabilities for learning with and without explicit reasoning. See details of additional models and experimental configurations in §B and examples of scratchpad in §A.
3.1 Testing the Limits of Transformers with Zero-shot, Few-shot and Finetuning
Limits of transformers in zero- and few-shot settings To investigate the inherent problem-solving capabilities of LLMs, we begin by analyzing models' zero-shot and few-shot performances on our compositional tasks. As shown in Figure 2, task performances deteriorate significantly from near perfection to zero with increasing complexity when measured by either problem size (Figure 2(a))or
Figure 3: GPT3 finetuned exhaustively on task-specific data up to a certain problem size. The blue region represents the in-distribution examples and the red region refers to OOD examples. The same trend is observed for the puzzle task (See §B.2)
Figure 4: Results of training beyond the overfitting point for the multiplication task with the goal of exploring whether OOD generalization capabilities (i.e., grokking) arise.
average parallelism (Figure 2(b)). The trend remains the same for few-shot prompting (see §B.2). These results indicate that pre-training is in fact not sufficient to teach models how to combine basic operations to solve compositional problems, especially as problems grow more complex.
Limits of transformers with question-answer training The limited performance of models may be attributed to the lack of task-specific data during pre-training. To fully bring out models' potentials in solving these tasks, we next exhaustively finetune GPT3 with question-answer pairs. In multiplication and DP, we finetune models with all enumerations of questions up to the maximum problem size $^4$  within reasonable training budget, leaving out  $10\%$  for validation and  $10\%$  for testing. In puzzles, we train on a subset of all instances up to  $(K,M)\leq (4,4)$  due to combinatorial explosion. We separately finetune GPT3 models on  $\sim 1.8\mathrm{M}$  multiplication pairs,  $\sim 142\mathrm{K}$  DP pairs, and  $\sim 41\mathrm{K}$  puzzle pairs (see details in §B.3). Additionally, to examine problems of different complexity, we consider different training splits based on the depth and width of computation graphs.
Figure 3 and Figure 5a show high accuracy for examples with splits seen during training, i.e., indomain. However, the performance sharply declines when evaluating unseen splits during training, i.e., out-of-domain (OOD). Similar trends hold in all tasks (see § B.3), suggesting that systematic problem-solving capabilities do not emerge via exhaustive training on task-specific data.
Limits of transformers with explicit scratchpad training Next, we test whether we can explicitly teach models the required computational operations via scratchpads. To do so, we finetune GPT3 with question-scratchpad pairs for all tasks. We consider the same distribution splits as before. The results, presented in Figure 5b, show that once again GPT3 achieves near-perfect performance on in-distribution, but fails entirely in generalizing to OOD cases—in particular, wider or deeper computation graphs. These results indicate that even when training directly with guidance on the computation steps, models still fail to learn component operations in a generalizable manner. This observation holds for all tasks (See details in § B.4). Similarly, prompting transformers with question-scratchpad pairs enhances the performance compared to the zero-shot setting (refer to § B.5). However, this performance boost diminishes to zero as complexity increases. These findings suggest that the autoregressive characteristic of transformers, which forces them to tackle problems sequentially, presents a fundamental challenge that cannot be resolved by instructing the model to generate a step-by-step solution. Instead, models depend on a greedy process of producing the next word to make predictions without a rigorous global understanding of the task.
Limits of transformers with grokking We explore whether extended training beyond overfitting leads to improved generalization abilities, a phenomenon known as grokking [59, 53]. Due to budget constraints, we only experiment on the multiplication task. Following [53], we fine-tune GPT3 with question-answer pairs for 420K steps and separately finetune GPT3 with question-scratchpad pairs for 30K steps. Both models' training far exceeded the point at which in-domain accuracy plateaus<sup>5</sup>. Figure 4 shows no improvement in generalization for OOD cases beyond the overfitting point, even
<sup>4</sup>We consider all  $k_{1}$ -by- $k_{2}$  digit multiplications with  $1 \leq k_{1}, k_{2} \leq 4$  and  $k_{1} \cdot k_{2} \leq 9$ ; and all DP problems up to 5 elements. We selected sizes based on budget constraints for GPT3 finetuning, see §B.3 for cost details.
5The training duration for question-answer pairs is equivalent to 60 epochs and costs 50,000 USD. Training on question-scratchpad pairs was conducted for 40 epochs and costs 40,000 USD.
(a) Results on question-answer pairs.
Figure 5: GPT3 finetuning and prompting accuracy on different data splits. Although the in-distribution performance is almost perfect, GPT3 exhibits poor generalization with increasing graph depth and width. Refer to §B.3 and §B.4 for results on the puzzle and DP tasks.
(b) Results on question-scratchpad pairs.
after extensive training periods. We hypothesize that the absence of grokking may be due to the level of difficulty of the task. We speculate that increased task difficulty significantly impedes learning a well-structured representation, which, according to [47], aligns with achieving grokking. Even if grokking were to emerge through more prolonged training, such an approach would prove inefficient and unscalable. Future work is required to accurately explain when and how grokking occurs.
3.2 Breaking Down Successes and Failures of Transformers
3.2.1 Information Gain Explains Where Transformers Partially Excel
At times transformers predict partially correct answers even when the overall response is incorrect. We speculate that this may be due to particularities in the task distribution that allow for guessing partial answers without performing the full multi-step reasoning that the task requires.
Using relative information gain (defined in §2.3), we can predict surface patterns that a model is likely to learn and contrast them empirically. For multiplication, relative information gain shows that the first digit (two digits) of the output highly correlates with the first digit (two digits) of each input number (see §C.1). Hence, this spurious pattern is likely to be learned by a model. Similarly, the prediction of the last digit (or two digits) of the output is observed to solely rely on the last digit (or two digits) of each input number. This pattern holds true due to the principles of modulo arithmetic, which ensures the validity of this relationship in all cases. Empirically, we verify that models indeed learn the patterns we predicted and other patterns as well (e.g., order of magnitude of the answer, number of trailing zeros for multiplication) in all the settings with and without scratchpad. See details for multiplication, plus dynamic programming task analysis in §C.
These experiments suggest that if an output element heavily relies on a single or a small set of input features, transformers are likely to recognize such correlation during training and directly map these input features to predict the output element in testing, without going through the rigorous multi-hop reasoning and giving a false illusion of performing compositional reasoning.
3.2.2 Transformers Reduce Multi-Step Compositional Reasoning into Linearized Subgraph Matching
We now explore whether models' correct predictions on unseen test data are due to learning the underlying algorithm or, instead, explainable by exposure to similar training examples. We hypothesize that, beyond simple memorization, transformers largely rely on pattern matching for solving these tasks. To test this, we calculate the average frequency with which partial computations needed to solve an instance appear in the training data, for both correctly and wrongly predicted examples.