SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Given a model-generated computation graph  $\widehat{G}_{A(\mathbf{x})}$  we analyze how often the full computation of each node  $v\in \widehat{V}$  is seen in training. We define  $v$  's full computation as the subgraph induced by all ancestors of  $v$  including  $v$ , denoted  $FC_{\widehat{G}_{A(\mathbf{x})}}(v)$ . We say that  $FC_{\widehat{G}_{A(\mathbf{x})}}(v)$  is seen during training if  $FC_{\widehat{G}_{A(\mathbf{x})}}(v)\cong FC_{G_{A(\mathbf{x}^{\prime})}}(w)$  for some computation graph  $G_{A(\mathbf{x}^{\prime})}$  in training, and for some  $w\in V$ . We characterize complexity of a full computation subgraph by its depth, as defined in §2.1.
Figure 6 shows that full computation subgraphs appear significantly more frequently in the training data for correctly predicted test examples than for incorrectly predicted ones, for both the multi-
Figure 6: Average frequency in which test examples' full computations subgraph appear in the training data w.r.t. the subgraph depth, grouped by final answer.
Figure 7: Ratio of nodes in each of the four correct/incorrect categories for each layer in computation graph. Results shown are for few-shot prompting and fine-tuning with scratchpad.
plication and DP task (both frequencies tend to zero for large depths since we ensured a disjoint train/test split). This high correlation suggests that pattern matching—and not general reasoning capabilities—may be the cause behind correct model outputs. This type of learning could be largely effective when the compositional complexity of tasks is low but it becomes less efficient when tasks are increasingly complex. This may elucidate the observed performance gain in low-complexity and in-domain cases and the striking performance drop in OOD and highly complex cases.
3.2.3 What Types of Errors do Transformers Make at Different Reasoning Depths?
For clearer understanding of where transformers fall short, we analyze the types of errors that transformers make for nodes at different layers in the computation graph. For every input  $\mathbf{x}$ , we compare the ground truth computation graph  $G_{A(\mathbf{x})}$  with the (possibly incorrect) model-generated computation graph  $\widehat{G}_{A(\mathbf{x})}$ . We consider a node  $v$  as having a correct value if and only if  $s(v) = \widehat{s}(v)$ . We consider a node  $v$  to be derived from a correct computation if given that  $U = \{u_1, \dots, u_k\}$  are the immediate predecessors of  $v$  in  $\widehat{G}_{A(\mathbf{x})}$  and that  $\widehat{o}p(v) = f$ , we have that  $f(u_1, \dots, u_k) = \widehat{s}(v)$ . Note that the notion of correct computation is independent of  $G$ , and that a node  $v$  derived from a correct computation may not have the correct value if an error occurred in some of its ancestors.
We classify each node  $v \in \widehat{V}$  into one of four categories. Node  $v$  is fully correct if  $v$  and its ancestors have correct values and are derived from correct computations. If a node  $v$  is not fully correct, its error can be of the following types:  $v$  has a local error if its parent nodes have correct values but  $v$  is derived from an incorrect computation (i.e., a one-hop reasoning error);  $v$  has a propagation error if  $v$  is derived from a correct computation but some of its parent nodes have incorrect values;  $v$  has a restoration error if it has a correct value but is derived from an incorrect computation.
Figure 7 shows results for few-shot GPT4 and fine-tuned GPT3 with scratchpad, with respect to graph layer number for each node. In all settings, the ratio of fully correct nodes is almost perfect but sharply decreases toward zero with increasing graph layers. Moreover, the ratio of propagation errors is usually higher than the ratio of local errors. Both phenomena suggest that models are able to correctly perform single-step reasoning, potentially due to memorizing such single-step operations during training, but fail to plan and compose several of these steps for an overall correct reasoning.
Both the DP and the puzzle tasks have a high ratio of restoration errors, suggesting memorization since correct outputs are produced despite incorrect computations. There are signs of memorization even when restoration errors are near zero:  $82.3\%$  of the final correct answers for 4-digit by 2-digit multiplications (a setting unseen during training) had at least one error in the computation graph, but still produced correct answers. These patterns are possibly due to high frequency of (input, output) multiplication pairs in the pretraining data, in contrast to intermediate reasoning steps.
If a node  $v$  does not appear in the ground truth graph  $G$ , we consider it to have an incorrect value.
4 Error Propagations: The Theoretical Limits
Experiments (§3) highlight the limitations of current transformers in handling complex, multi-step reasoning tasks. Concretely, we show that errors rapidly escalate as the problem size grows (§3.2.3). Here, we aim to provide theoretical insights into why autoregressive transformer LLMs can perform significantly worse in compositional tasks as the problem size increases, making explicit the different ways in which compounding stochastic errors affect final performance. We argue using stylized examples that transformers may be too limited to solve compositionally complex tasks. Formal statements and full proofs are provided in §D.
Algorithms designed to solve compositional tasks typically involve multiple independent applications of a function and/or iterated applications of the same function. A transformer executing such an algorithm acts as an estimator of these functions. In this context, we examine the probability of such an estimator reaching the correct answer as the problem size increases. We first consider a scenario where a transformer estimates an algorithm requiring  $n$  independent applications of a function:
Proposition 4.1 (informal). Let  $f_{n}$  involve the combination  $h_n$  of  $n$  independent applications of a function  $g$ . Let  $\widehat{f}, \widehat{g}, \widehat{h}_n$  be their estimators. Assume that  $\widehat{h}_n$  is a perfect estimator of  $h_n$  and that  $h_n$  has low collision, with  $c_n$  being an upper bound of  $h_n$ 's collision rate ( $c_n < c \forall n$ , with  $c \ll 1$ ). If  $\mathbb{P}(g \neq \widehat{g}) = \epsilon > 0$  where  $\widehat{g}$ 's errors are independent, then  $\mathbb{P}(f_n \neq \widehat{f}_n) > 1 - c_n - (1 - \epsilon)^n \cdot (1 - c_n)$ . This implies that  $\mathbb{P}(f_n \neq \widehat{f}_n)$  decreases exponentially as  $n$  increases, with  $\lim_{n \to +\infty} \mathbb{P}(f_n \neq \widehat{f}_n) \geq 1 - c$ . Moreover, if  $c_n \leq \beta \alpha^n$  for some  $\alpha \in (0, 1)$ ,  $\beta > 0$ ,  $\mathbb{P}(f_n \neq \widehat{f}_n)$  tends exponentially to 1 as  $n$  increases.
Prop. 4.1's proof ( $\S$ D.1) shows the rate of convergence is exponential, thus concluding that transformers will rapidly fail with increasing  $n$ . Let's now analyze the iterated application function scenario.
Proposition 4.2 (informal). Let  $f_{n}(\mathbf{x}) = g^{n}(\mathbf{x})$  involve the repeated application of  $g$ . Assume that the probability of recovering from a mistake due to the randomness of applying the estimator on an incorrect input has probability at most  $c$ . If  $\mathbb{P}(g \neq \widehat{g}) = \epsilon > 0$ , then  $\mathbb{P}(f_n \neq \widehat{f}_n)$  decreases exponentially with  $n$ . Precisely,  $\mathbb{P}(f_n \neq \widehat{f}_n) \geq 1 - (1 - \epsilon - c)^{n-1} (1 - \epsilon - c / (c + \epsilon))$ , implying  $\lim_{n \to +\infty} \mathbb{P}(f_n \neq \widehat{f}_n) \geq 1 - c / (c + \epsilon)$ .
The argument is as follows. Let  $s_n \coloneqq \mathbb{P}(f_n = \widehat{f}_n)$ , where  $s_1 = 1 - \epsilon$  by definition. Derive  $s_n \leq (1 - \epsilon - c) \cdot s_{n-1} + c$  using law of total probability. Then, prove by induction a non-recursive upper bound for  $s_n$  with limit  $\frac{c}{c + \epsilon}$  when  $n \to +\infty$ . See formal statement and derivation in §D.2.
Prop. 4.2's proof also shows an exponential rate of convergence. Note that if  $c \ll \epsilon$  then  $\lim_{n \to +\infty} \mathbb{P}(f_n \neq \widehat{f}_n) \approx 1$ . It is reasonable to assume  $c \ll \epsilon$  when  $g$  has low collision, since  $c$  represents the probability of the estimator  $\widehat{g}(y)$  arriving at the correct output  $g(x)$  by chance when given the wrong input  $y \neq x$ . More details in §D.3.
Moreover, repeated applications of a function often imply unbounded errors: if  $g(x)$  can be expressed as an affine transformation  $Fx + c$ , then it may be viewed as a first-order vector autoregression, which are known to be unstable when  $|\lambda| \geq 1$  for at least one  $\lambda$  eigenvalue of  $F$  [31, Prop. 10.1]. While we make these arguments with affine maps, similar behaviors, possibly even more acute, could occur with nonlinear maps [25]—but their study is beyond the scope of this paper.
In Prop. 4.2's current form, we implicitly assume that there is a single valid reasoning for each input since  $g$  is a function. We can potentially generalize this assumption with a state-transition framing, where the probability of transitioning from a valid state to an invalid one is  $\epsilon$ , and the probability of recovering from an invalid state is at most  $c$ . See formal statement in D.2.
All tasks evaluated in the present work can be seen as instances of the results just proven. Prop. 4.1 directly applies to multiplication, since  $m$ -by- $n$  digit multiplication can be seen as  $n$  independent instances of  $m$ -by-1 digit multiplication (see Cor. D.1). Prop. 4.2 directly applies to the recursive function of the dynamic programming task, as well as to  $m$ -by-1 digit multiplication, and to the puzzle through its elimination function (details in D.3). They are also all low collision settings.
Note that Prop 4.1 and 4.2 apply to any high-performant estimator of reasoning tasks. We focus on out-of-the-box transformers to align with the scope of our experiments and with the goal of framing empirical results. In §5, we discuss how these propositions may inform future research directions.
Collapsed Compositionality and Robustness Implications Transformers today demonstrate undeniably powerful empirical results. Yet, our study suggests that they may have fundamental weaknesses in certain intellectual tasks that require true multi-step compositional operations such as multiplications and logic puzzles. Our careful study based on the computation graph and analyses demonstrates that transformers can often solve multi-step compositional problems by collapsing the depth of the compositional operations via analogical pattern matching. More broadly, our findings suggest that the strong performance of transformers should be taken with a certain grain of salt: Despite initially appearing challenging, certain tasks may not possess the inherent compositionality they seem to have. This is due to the fact that desired solutions could be readily derived from input-output sequences present in the training data, allowing for shortcut pattern matching to produce acceptable solutions. However, such an approach can ultimately result in poor generalization as shown in our study. For example, fine-tuning GPT3 on our tasks both with and without explicit reasoning graphs shows that models' learning fails to generalize beyond levels of complexity seen in training.
Theoretical Findings and their Empirical Implications The proofs presented in §4 show that, under reasonable assumptions, the probability of incorrect predictions converges exponentially to  $\approx 1$  for abstract compositional tasks. Importantly, these proofs apply to autoregressive LMs in general. Our insights indicate that the current configuration of transformers, with their reliance on a greedy process for predicting the next word, constrains their error recovery capability and impedes the development of a comprehensive global understanding of the task. Building on these findings, we suggest several empirical strategies for harnessing the potential of transformers. Firstly, transformers may be employed in ways that require chaining only a few compositional steps to reach a solution rather than lengthy reasoning steps (e.g., [35]). Secondly, transformers may be best suited for compositional tasks where evaluation metrics can afford some leniency; for example, finding approximate solutions that do not require executing the whole graph, such as identifying the most significant digit in a multiplication. Finally, we suggest augmenting transformers with planning modules as well as using refinement methods, that can iteratively improve their generations [82, 48].
Call for Broad Participation to Investigate Limitations Identification of limitations is an important step towards achieving greater robustness. Our study suggests fundamental limitations that impede transformers from fully mastering certain compositional operations. However, we acknowledge that due to our compute budget constraints as well as limited access to the largest language models such as GPT4, we are unable to push the empirical limits of transformers even further in terms of training data size and number of epochs. We invite the broader research community, particularly those with more extensive resources at their disposal, to investigate these possibilities further.
Reasoning abilities in transformer LLMs Recently, transformers [11, 58, 57, 17, 16, 63, 73, 74] have demonstrated impressive reasoning abilities across a wide range of tasks, even outperforming humans in certain cases [79, 28, 15, 85]. This success has been largely attributed to the scaling effect, where larger models and training datasets result in improved performance [38, 33, 1]. However, these models have also been shown to struggle across multiple domains [32], including algorithmic reasoning [78], commonsense reasoning [62, 40], theory of mind [65], planning [76], logical reasoning [70], and ethical reasoning [36]. These difficulties have motivated us to take a step back and thoroughly examine both the successes and failures of transformers from empirical and theoretical perspectives on compositional reasoning tasks.
Challenges of transformers in compositional tasks Transformers perform fairly well in single-step reasoning tasks [70], but face challenges when it comes to effectively combining multiple steps to solve compositionally complex problems [84, 55, 66, 81]. Recent research has focused on overcoming these limitations through various approaches. First, fine-tuning transformers to directly generate the final answer while keeping the reasoning implicit [7, 18]. Second, encouraging transformers to generate reasoning steps explicitly within a single generation [55, 80, 44, 42]. For example, Nye et al. [55] and Zhou et al. [86] used scratchpads to teach transformers how to perform algorithmic reasoning tasks such as addition by splitting the task into intermediate steps [44, 80]. Further, leveraging LLMs to generate each reasoning step iteratively via a selection and inference mechanism [20, 19, 72].
Lastly, choosing a training split that maximizes the number of observed patterns between the train and test data [10], or diversifying in-prompt examples to cover the maximum of patterns [41], ultimately enhancing generalization. The primary focus of these studies is to enhance model performance on compositional problems without striving for complete mastery. In contrast, our work explores the fundamental limits of vanilla transformers in achieving full mastery, striving for  $100\%$  performance in both in-domain and OOD settings. Our findings show that reaching full mastery is inherently challenging, providing insights into the complexities involved.
Challenges of transformers in generalization Extensive research has been done to investigate the generalization capabilities of transformers [3, 54, 26, 64, 8, 46]. This encompasses various facets of generalization, including easy-to-hard generalization [67, 4], length generalization [2, 60, 13, 54, 8], and generalization on symbolic mathematical integration [83]. Schwarzschild et al. [67] and Bansal et al. [4] employ weight-tied neural networks to generalize from easy to hard examples. Liu et al., [45] found that shallow transformers learn shortcuts during training, leading to poor OOD generalization. Razeghi et al. [64] revealed a positive correlation between the frequency of training terms and their test performance. Building upon this line of inquiry, we present a more rigorous examination of sub-graph matching between training and test instances for complex compositional tasks where we demonstrate how pattern matching can hinder generalization. We complement our empirical results with theoretical insights on transformers' limits.
Grokking The phenomena of models' gaining generalization capabilities when training significantly beyond overfitting, known as grokking was recently introduced in [59]. Subsequent works focus on characterizing when and why grokking arises: [47] show that perfect generalization in an arithmetic addition task happens when there is sufficient data to determine the appropriate structured representation, later extended to sparse parity in [52] where a sparse subnetwork of neurons is shown responsible for generalization behavior. Recently, [77] propose that grokking occurs when a task admits a generalizing and a memorizing solution, and the former is slower to learn. In this present work, our aim is not to explain grokking but rather to observe its emergence. We do not observe grokking arising in the context of multiplication, and we leave it to future work to explore whether this may be due to task difficulty hindering the learning of well-structured representations.
Transformers' theoretical expressiveness Lin et al. [43] study autoregressive models' limitations from a computational complexity theory perspective. Transformer-specific work has focused on quantifying the class of problems that (not necessarily autoregressive) transformers can express assuming perfect parameters [51, 50, 14, 49, inter alia]. All tasks analyzed in our work belong to a class expressible by transformers, suggesting that known upper bound might not be tight. Importantly, Hahn [30] shows that transformers cannot robustly model noncounter-free regular languages even when allowing infinite precision. In contrast, our focus is on error accumulation, which enables to investigate if reasoning tasks theoretically solvable by transformers are likely to be solved by them.
Additional literature and societal impact discussion can be found in  $\S E$
On a broader scope, as transformers continue to gain widespread deployment with significant real-world impacts, it is ever more urgent to understand their successes and failures. Our study critically investigates transformers' limitations and emphasizes the need to develop models capable of robust generalization and systematic problem-solving. By examining the compositional capabilities of these models, we aspire to work towards more reliable AI systems that excel not only in tasks where abundant training examples are sufficient, but also in cases requiring precise compositional reasoning.
We focus on analyzing compositional reasoning capabilities through the lens of computation graphs. Although they are a useful way to systematically represent rigorous reasoning processes, it is important to note that for the scratchpad approach, we are limited to only establishing a correlation between the model generation and its preceding context, as we cannot inspect the exact tokens model attends to when making the prediction. This limitation arises from our lack of access to the activations of the studied models. Furthermore, we posit that alternative approaches to linearizing reasoning processes may yield different performances and provide opportunities for further exploration.
We thank members of the Mosaic team at AI2 for valuable feedback on this project, as well as Agustín Santiago Gutiérrez and Kawin Ethayarajh for valuable discussions. This research was supported by the NSF DMS-2134012, DARPA MCS program through NIWC Pacific (N66001-19-2-4031), and the Allen Institute for AI.
A Compositional Tasks 19
A.1 Multiplication 19
A.2 Einstein's Puzzle 20
A.3 Dynamic Programming Problem 22
B Experimental Setups & Empirical Results 24
B.1 Models 24
B.2 Limits of Transformers in Zero- and Few-shot Settings 24
B.3 Limits of Transformers with question-answer Training 24
B.4 Limits of Transformers with Explicit Scratchpad Training 26
B.5 Limits of Transformers with Explicit Scratchpad Prompting 26
C Surface Patterns 29
C.1 Relative Information Gain Predictions for Multiplication 29
C.2 Empirical Surface Pattern Analysis for Multiplication with GPT4, ChatGPT and GPT3 30
C.3 Relative Information Gain Predictions for Dynamic Programming Task 31
C.4 Empirical Surface Pattern Results for Dynamic Programming Task 33
D Theoretical Results: Derivations 35
D.1 Error accumulates with larger parallel applications of an estimated function (width) 35
D.2 Error accumulates with larger iterative applications of an estimated function (depth) 36
D.3 Discussing  $c\ll \epsilon$  in the context of Proposition 4.2 38
E Additional Literature and Societal Impact 40
E.1 Additional Literature 40
E.2 Societal Impact Discussion 40
A Compositional Tasks
A.1 Multiplication
Data Construction We exhaustively generate multiplication problems as question-answer pairs (e.g., Q: "What is 4 times 32?" A: "128"). We focus on multiplications of two numbers  $x = (x_{1}, x_{2}, \ldots, x_{k})$  and  $y = (y_{1}, y_{2}, \ldots, y_{k})$  where each number can have up to  $k$  digits, amounting to  $9 \times 10^{(k-1)}$  combinations per each number. We set  $k$  to 5 in our experiments. Figure 8 showcases an example prompt for performing few-shot learning without the inclusion of a scratchpad, while Figure 9 demonstrates an example prompt using a scratchpad. Throughout our experimentation, we explored various versions of the scratchpad, ranging from verbose and detailed to more concise alternatives. Among these variations, the scratchpad version depicted in Figure 9 ultimately produced the most favorable outcomes. Listing 1 shows the Python code for solving the task.
To multiply two numbers, start by multiplying the rightmost digit of the multiplicand by each digit of the multiplier, writing down the products and carrying over any remainders. Repeat this process for each digit of the multiplicand, and then add up all the partial products to obtain the final result.
Questions: what's 22 times 2? Answer 44.
Figure 8: Example prompt for the multiplication task used for the few-shot setting.
Question: What is 35 times 90?
Scratchpad: Let's perform the multiplication step by step:
Let's multiply 35 by the digit in the ones place of 90, which is 0.
1. Multiply 0 by the digit in the ones place of 35, which is 5. This gives  $5 \times 0 = 0$ . Write down the result 0.
2. Multiply 0 by the digit in the tens place of 35, which is 3. This gives  $3 \times 0 = 0$ . Write down the result 0.
3. The partial product for this step is  $A = 0$  which is the concatenation of the digits we found in each step.
Now, let's multiply 35 by the digit in the tens place of 90, which is 9.
4. Multiply 9 by the digit in the ones place of 35, which is 5. This gives  $5 \times 9 = 45$ . Write down the result 5 and carry over the 4 to the next step.
5. Multiply 9 by the digit in the tens place of 35, which is 3. Add the carryover from the previous step to account for this. This gives  $(3 \times 9) + 4 = 31$ . Write down the result 31.
6. The partial product for this step is  $B = 315$  which is the concatenation of the digits we found in each step.
Now, let's sum the 2 partial products A and B, and take into account the position of each digit: A=0 (from multiplication by 0) and B=315 (from multiplication by 9 but shifted one place to the left, so it becomes 3150). The final answer is 0 x 1 + 315 x 10 = 0 + 3150 = 3150.
Figure 9: A sample scratchpad for the multiplication task.
1 def multiply(x, y):  
2 summands = [0] * len(y)  
3 for i in range(len(y) - 1, -1, -1):